Kinect Dawanoid: How to create a ball game controlled only with ambient noise

During a French event called TechDays 2013, I presented a session called “Coding4Fun”. And during this session, I demoed a game called Kinect Dawanoid.

The goal of Kinect Dawanoid is to control a paddle (the racket of the Arkanoid game) with ambient noise only.

The big deal is that there is not one player but hundreds . In this case, the paddle will go to the left if there is more noise to the left and obviously the paddle will go to the right if there is more noise to the right:

Infographic courtesy of Michel Rousseau

 

To well understand how it looks like, I suggest you to have a look at this replay: 

The complete session (in french sorry):   
https://www.youtube.com/watch?v=wTtnhal35Rc&feature=youtu.be / https://www.microsoft.com/france/mstechdays/

The game itself

Before looking at the Kinect part, let me introduce you with the concept (!!) of the game:

 

The principle is simple: You control the paddle (until now with cursors keys) and the game uses a really simple physic system to simulate the movement of a ball. You lose the game when the ball falls down.

For the sake of clarity, I did not include the bricks.

Kinect Dawanoid is a simple WPF application. To control the ball, you need the following variables:

Point ballPosition = new Point(0, 0);
Vector ballDirection = new Vector(2, 2);

The game itself is based on two functions:

void StartGame()
{
    CompositionTarget.Rendering += CompositionTarget_Rendering;
    KeyDown += MainWindow_KeyDown;
}

The paddle is controlled using the _MainWindow_KeyDown_ method:

void MainWindow_KeyDown(object sender, KeyEventArgs e)
{
    switch (e.Key)
    {
        case Key.Left:
            padDirection -= padSpeed;
            break;
        case Key.Right:
            padDirection += padSpeed;
            break;
    }
}

The rendering part is based on the _CompositionTarget_Rendering_ method:

void CompositionTarget_Rendering(object sender, EventArgs e)
{
    // Pad
    padPosition += padDirection;
    if (padPosition < 0)
    {
        padDirection = 0;
        padPosition *= -1;
    }
    else if (padPosition > playground.RenderSize.Width - pad.Width - 1)
    {
        padPosition = playground.RenderSize.Width - pad.Width - 1;
        padDirection *= -1;
    }

    padDirection *= padInertia;


    // Ball
    ballPosition += ballDirection;

    // Walls
    if (ballPosition.X < 0)
    {
        ballPosition.X = 0;
        ballDirection.X *= -1;
    }
    else if (ballPosition.X >= playground.RenderSize.Width - ball.Width)
    {
        ballPosition.X = playground.RenderSize.Width - ball.Width - 1;
        ballDirection.X *= -1;
    }

    if (ballPosition.Y < 0)
    {
        ballPosition.Y = 0; 
        ballDirection.Y *= -1;
    }
    else if (ballPosition.Y >= playground.RenderSize.Height - ball.Height)
    {
        CompositionTarget.Rendering -= CompositionTarget_Rendering;
        ScoreText.Text = "Final score: " + (score / 10).ToString();
        return;
    }

    // Collisions
    var padRect = new Rect(padPosition, playground.RenderSize.Height - 50, pad.Width, pad.Height);
    var ballRect = new Rect(ballPosition, new Size(ball.Width, ball.Height));

    if (padRect.IntersectsWith(ballRect))
    {
        ballPosition.Y = playground.RenderSize.Height - 50 - ball.Height;
        ballDirection.Y *= -1;
    }

    // Moving
    Canvas.SetLeft(ball, ballPosition.X);
    Canvas.SetTop(ball, ballPosition.Y);

    Canvas.SetTop(pad, playground.RenderSize.Height - 50);
    Canvas.SetLeft(pad, padPosition);

    // Score
    ScoreText.Text = (score / 10).ToString();
    score++;
}

Nothing special here. The system evaluates the new ball position, check the collisions with the pad and the walls and finally update the position of the ball (which is represented by a XAML ellipse Inside a canvas) and the paddle (a rectangle).

 

Controlling the paddle with ambient noise

Thanks to Kinect, we have a wonderful way to detect ambient noise (via beam angle).

Indeed, to capture audio, the Kinect sensor receives sounds from every direction. However, like the cone of light from a lighthouse where the light is the brightest, the audio capture hardware has an imaginary cone that is able to capture audio signals the best. Audio waves that propagate through the length of the cone can be separated from audio waves that travel across the cone. If you point the cone in the direction of the audio that your application is most interested in capturing, you can improve the ability to capture and separate that audio source from other competing audio sources.

Similar to the sound source angle, the beam angle is also defined in the x-z plane of the sensor perpendicular to the z-axis. The beam angle and the sound source angle are both updated continuously once the sensor has started streaming audio data (when the Start method is called).

Furthermore, Kinect SDK provides you with an event raised when beam angle changes. So, using the following code, you can easily track the position of the strongest sound source:

 

using Microsoft.Kinect;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Dawanoïd
{
    public class BeamManager : INotifyPropertyChanged
    {
        readonly KinectAudioSource audioSource;

        public BeamManager(KinectAudioSource source)
        {
            audioSource = source;
            audioSource.BeamAngleChanged += audioSource_BeamAngleChanged;
        }

        void audioSource_BeamAngleChanged(object sender, BeamAngleChangedEventArgs e)
        {
            if (PropertyChanged != null)
            {
                PropertyChanged(this, new PropertyChangedEventArgs("BeamAngle"));
            }
        }

        public double BeamAngle
        {
            get
            {
                return Math.Max(0, Math.Min(1, (audioSource.BeamAngle - KinectAudioSource.MinBeamAngle)
                            / (KinectAudioSource.MaxBeamAngle - KinectAudioSource.MinBeamAngle)));
            }
        }

        public void Dispose()
        {
            audioSource.BeamAngleChanged -= audioSource_BeamAngleChanged;
        }

        public event PropertyChangedEventHandler PropertyChanged;
    }
}

 

So instead of controlling the paddle with the keyboard, we can just add this line in the _CompositionTarger_Rendering_ method:

padDirection += beamManager.BeamAngle * 2.0 - 1.0;

And voila!!

The best way to understand Kinect Dawanoïd is to try it: https://www.catuhe.com/msdn/dawanoïd.zip