Dev

KISS : Use Kinect for Windows SDK to protect your privacy

Fotolia_27381572_S

I have the incredible chance to work with a very fun and motivating team. But one drawback of this situation is the risk to send unwanted emails when you forget to lock your computer when you go out.

And they can be really creative with this so French and sophisticated humor when they use your Outlook. So to protect myself I decided to write a small program using one of my favorite technology : Kinect for Windows SDK.

The main goal of KISS (which stands for Kinect Intelligent Security System) is to track the user in front of the sensor and detect when he goes out (to lock the computer for example).

You will discover how to develop such a program during this article.

The final solution is available to download there: https://www.catuhe.com/msdn/kiss.zip

 

Initializing the sensor

First of all, you have to reference the Kinect for Windows SDK assembly (C:Program FilesMicrosoft SDKsKinectv1.0AssembliesMicrosoft.Kinect.dll) and instantiate a new KinectSensor. To do so, the SDK helps you with a static helper class called KinectSensor which contains a list of already detected sensors : KinectSensor.KinectSensors. It also provides an event KinectSensor.KinectSensors.StatusChanged which will be raised when something happens to one of the attached Kinect sensors.






  1. public KinectIntelligentSecuritySystem()


  2. {


  3. try


  4. {


  5. //listen to any status change for Kinects


  6. KinectSensor.KinectSensors.StatusChanged += Kinects_StatusChanged;


  7.  


  8. //loop through all the Kinects attached to this PC, and start the first that is connected without an error.


  9. foreach (KinectSensor kinect in KinectSensor.KinectSensors)


  10. {


  11. if (kinect.Status == KinectStatus.Connected)


  12. {


  13. kinectSensor = kinect;


  14. break;


  15. }


  16. }


  17.  


  18. if (kinectSensor == null)


  19. MessageBox.Show(“No Kinect found”);


  20. else


  21. Initialize();


  22.  


  23. }


  24. catch (Exception ex)


  25. {


  26. MessageBox.Show(ex.Message);


  27. }


  28. }




 






  1. void Kinects_StatusChanged(object sender, StatusChangedEventArgs e)


  2. {


  3. switch (e.Status)


  4. {


  5. case KinectStatus.Connected:


  6. if (kinectSensor == null)


  7. {


  8. kinectSensor = e.Sensor;


  9. Initialize();


  10. }


  11. break;


  12. case KinectStatus.Disconnected:


  13. if (kinectSensor == e.Sensor)


  14. {


  15. Clean();


  16. MessageBox.Show(“Kinect was disconnected”);


  17. }


  18. break;


  19. case KinectStatus.NotPowered:


  20. if (kinectSensor == e.Sensor)


  21. {


  22. Clean();


  23. MessageBox.Show(“Kinect is no more powered”);


  24. }


  25. break;


  26. }


  27. }




 

As you can see, it is pretty simple as you just have to track the StatusChanged event and call Clean or Initialize accordingly:






  1. private void Initialize()


  2. {


  3. if (kinectSensor == null)


  4. return;


  5.  


  6. kinectSensor.Start();


  7.  


  8. kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution80x60Fps30);


  9. kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;


  10. kinectSensor.DepthStream.Range = DepthRange.Near;


  11. }


  12.  


  13. void Clean()


  14. {


  15. kinectSensor.Stop();


  16. kinectSensor = null;


  17. }




A thing worth noting is the activation of the near mode:






  1. kinectSensor.DepthStream.Range = DepthRange.Near;




With this mode activated, skeletons tracking is disabled but the depth values can be retrieved from a near distance (about 40cm).

 

Displaying the depth stream

Now, the sensor is instantiated and ready to use. You have then to connect to the depth stream.

When you develop with Kinect, it is important to provide a visual feedback of what the sensor sees. An event is already handled in the Initialize method to do get the depth frames:






  1. kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;




So you have to create a method that can build a bitmap from the depth stream:






  1. void DisplayData(DepthImageFrame frame)


  2. {


  3. for (int i16 = 0, i32 = 0; i16 < data.Length && i32 < data32.Length; i16++, i32++)


  4. {


  5. int realDepth = (data[i16] >> 3);


  6. byte intensity = (byte)(255 – (255 realDepth / 3000.0f));


  7.  


  8. data32[i32] = (intensity / 2) + ((intensity / 2) << 16) + ((intensity / 2) << 8) + (255 << 24);


  9. }


  10.  


  11. if (DepthBitmap == null)


  12. {


  13. DepthBitmap = new WriteableBitmap(frame.Width, frame.Height, 96, 96, PixelFormats.Bgra32, null);


  14. }


  15.  


  16. DepthBitmap.Lock();


  17.  


  18. int stride = DepthBitmap.PixelWidth DepthBitmap.Format.BitsPerPixel / 8;


  19. Int32Rect dirtyRect = new Int32Rect(0, 0, DepthBitmap.PixelWidth, DepthBitmap.PixelHeight);


  20. DepthBitmap.WritePixels(dirtyRect, data32, stride, 0);


  21.  


  22. DepthBitmap.AddDirtyRect(dirtyRect);


  23. DepthBitmap.Unlock();


  24.  


  25. if (PropertyChanged != null)


  26. {


  27. PropertyChanged(this, new PropertyChangedEventArgs(“DepthBitmap”));


  28. }


  29. }




DisplayData uses a DepthFrame (provided by the event) to fill a WriteableBitmap with grayed values (from white (near) to black (far)).

The KinectIntelligentSecuritySystem class implements INotifyPropertyChanged and exposes a property called DepthBitmap. This property is used by the main program to fill a WPF Image control:






  1. <Window x:Class=”KISS.MainWindow”


  2. xmlns=”https://schemas.microsoft.com/winfx/2006/xaml/presentation"


  3. xmlns:x=”https://schemas.microsoft.com/winfx/2006/xaml"


  4. Title=”Kinect Intelligent Security System” Height=”350” Width=”525”>


  5. <Grid>


  6. <Image x:Name=”depthImage” Source=”{Binding DepthBitmap}“ />


  7. </Grid>


  8. </Window>




 






  1. depthImage.DataContext = kiss;




So every time DisplayData computes a new image, the PropertyChanged event is raised and the WPF Image control updates itself.

This code is directly inspired by the Kinect Toolbox that you can grab there : https://kinecttoolbox.codeplex.com

 

Detecting user and locking the computer

Finally the main point is here: using the depth stream to compute the average depth of what is in front of the sensor and to detect any big variation of this average:






  1. // Computing depth average


  2. var avg = data.Average(pixel => pixel);


  3.  


  4. previousValues.Add(avg);


  5.  


  6. var currentAvg = previousValues.Average(value => value);


  7.  


  8. if (previousValues.Count > 60)


  9. previousValues.RemoveAt(0);


  10.  


  11. if (previousValues.Count == 60 && (Math.Abs(currentAvg – avg) > 1500))


  12. {


  13. if (OnMovement != null)


  14. OnMovement();


  15.  


  16. previousValues.Clear();


  17. }




You have to save a given number (60 in this example) of previous computed average depths in the previousValues list. With this list, every time a new frame is available, you have to compare the current average with the average of values stored in the list and if the difference is bigger than a given threshold, an event is raised.

In response to this event, I choose to lock my computer with this simple interop code:






  1. readonly KinectIntelligentSecuritySystem kiss = new KinectIntelligentSecuritySystem();


  2.  


  3. [DllImport(“user32.dll”)]


  4. public static extern void LockWorkStation();


  5.  


  6. public MainWindow()


  7. {


  8. InitializeComponent();


  9. kiss.OnMovement += kiss_OnMovement;


  10.  


  11. depthImage.DataContext = kiss;


  12. }


  13.  


  14. void kiss_OnMovement()


  15. {


  16. LockWorkStation();


  17. }




And voila! You are now safe to get out without locking your computer because KISS watches over you Rire.

The complete code for KISS is the following:






  1. using System;


  2. using System.Linq;


  3. using System.Windows;


  4. using System.Windows.Media;


  5. using System.Windows.Media.Imaging;


  6. using Microsoft.Kinect;


  7. using System.Collections.Generic;


  8. using System.ComponentModel;


  9.  


  10. namespace KISS


  11. {


  12. class KinectIntelligentSecuritySystem : INotifyPropertyChanged


  13. {


  14. public event Action OnMovement;


  15. public event PropertyChangedEventHandler PropertyChanged;


  16.  


  17. public WriteableBitmap DepthBitmap { get; private set; }


  18.  


  19. private KinectSensor kinectSensor;


  20. readonly List<double> previousValues = new List<double>();


  21. short[] data;


  22. int[] data32;


  23.  


  24. void Kinects_StatusChanged(object sender, StatusChangedEventArgs e)


  25. {


  26. switch (e.Status)


  27. {


  28. case KinectStatus.Connected:


  29. if (kinectSensor == null)


  30. {


  31. kinectSensor = e.Sensor;


  32. Initialize();


  33. }


  34. break;


  35. case KinectStatus.Disconnected:


  36. if (kinectSensor == e.Sensor)


  37. {


  38. Clean();


  39. MessageBox.Show(“Kinect was disconnected”);


  40. }


  41. break;


  42. case KinectStatus.NotPowered:


  43. if (kinectSensor == e.Sensor)


  44. {


  45. Clean();


  46. MessageBox.Show(“Kinect is no more powered”);


  47. }


  48. break;


  49. }


  50. }


  51.  


  52. public KinectIntelligentSecuritySystem()


  53. {


  54. try


  55. {


  56. //listen to any status change for Kinects


  57. KinectSensor.KinectSensors.StatusChanged += Kinects_StatusChanged;


  58.  


  59. //loop through all the Kinects attached to this PC, and start the first that is connected without an error.


  60. foreach (KinectSensor kinect in KinectSensor.KinectSensors)


  61. {


  62. if (kinect.Status == KinectStatus.Connected)


  63. {


  64. kinectSensor = kinect;


  65. break;


  66. }


  67. }


  68.  


  69. if (kinectSensor == null)


  70. MessageBox.Show(“No Kinect found”);


  71. else


  72. Initialize();


  73.  


  74. }


  75. catch (Exception ex)


  76. {


  77. MessageBox.Show(ex.Message);


  78. }


  79. }


  80.  


  81. private void Initialize()


  82. {


  83. if (kinectSensor == null)


  84. return;


  85.  


  86. kinectSensor.Start();


  87.  


  88. kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution80x60Fps30);


  89. kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;


  90. kinectSensor.DepthStream.Range = DepthRange.Near;


  91. }


  92.  


  93. void Clean()


  94. {


  95. kinectSensor.Stop();


  96. kinectSensor = null;


  97. }


  98.  


  99. void DisplayData(DepthImageFrame frame)


  100. {


  101. for (int i16 = 0, i32 = 0; i16 < data.Length && i32 < data32.Length; i16++, i32++)


  102. {


  103. int realDepth = (data[i16] >> 3);


  104. byte intensity = (byte)(255 – (255 realDepth / 3000.0f));


  105.  


  106. data32[i32] = (intensity / 2) + ((intensity / 2) << 16) + ((intensity / 2) << 8) + (255 << 24);


  107. }


  108.  


  109. if (DepthBitmap == null)


  110. {


  111. DepthBitmap = new WriteableBitmap(frame.Width, frame.Height, 96, 96, PixelFormats.Bgra32, null);


  112. }


  113.  


  114. DepthBitmap.Lock();


  115.  


  116. int stride = DepthBitmap.PixelWidth DepthBitmap.Format.BitsPerPixel / 8;


  117. Int32Rect dirtyRect = new Int32Rect(0, 0, DepthBitmap.PixelWidth, DepthBitmap.PixelHeight);


  118. DepthBitmap.WritePixels(dirtyRect, data32, stride, 0);


  119.  


  120. DepthBitmap.AddDirtyRect(dirtyRect);


  121. DepthBitmap.Unlock();


  122.  


  123. if (PropertyChanged != null)


  124. {


  125. PropertyChanged(this, new PropertyChangedEventArgs(“DepthBitmap”));


  126. }


  127. }


  128.  


  129. void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)


  130. {


  131. var frame = e.OpenDepthImageFrame();


  132.  


  133. if (frame == null)


  134. return;


  135.  


  136. if (data == null)


  137. {


  138. data = new short[frame.PixelDataLength];


  139. data32 = new int[frame.PixelDataLength];


  140. }


  141.  


  142. frame.CopyPixelDataTo(data);





  144. // Displaying frame


  145. DisplayData(frame);


  146.  


  147. frame.Dispose();





  149. // Computing depth average


  150. var avg = data.Average(pixel => pixel);


  151.  


  152. previousValues.Add(avg);


  153.  


  154. var currentAvg = previousValues.Average(value => value);


  155.  


  156. if (previousValues.Count > 60)


  157. previousValues.RemoveAt(0);


  158.  


  159. if (previousValues.Count == 60 && (Math.Abs(currentAvg – avg) > 1500))


  160. {


  161. if (OnMovement != null)


  162. OnMovement();


  163.  


  164. previousValues.Clear();


  165. }


  166. }


  167.  


  168. }


  169. }