(For more information consult the JGameGrid web site.)

NEW! Sound Direction Finder using the Kinect microphone array

The Kinect device

The Kinect controller is a natural user interface (NUI) that can track persons by a depth sensing video camera and generate skeletal information for 20 body joints in real-time. It also contains a microphone array to capture sound with direction information. There are numerous potential uses including game control, robotics, surveillance, input device for exhibition stands, people tracking, etc.

In 2010 a non-official Kinect driver with a C++ API became available and is now distributed through Code Laboratories and OpenNI. There are also some Java wrappers, especially with Greenfoot, but the installation and use is pretty tricky. In 2011 Microsoft released an official development system for the Kinect device called Kinect SDK. It's fully based on the Visual Studio IDE and all examples are in C++ and C#. As usual with Microsoft, it is restricted to the Windows OS and runs only under Windows 7 and higher. As soon as this official release became available we decided to create a simple Java wrapper library KinectJLib based on our Java API Wrapper (JAW).

How it works

JAW is an object oriented communication framework between Java and native C++ code based on the Java Native Interface (JNI). It provides access to the C based Windows API, can create native threads and handles native windows through the standard windows message loop. Because the Kinect SDK contains examples in native C++ code, it is straightforward to reuse this code in a Java wrapper based on JAW. The native video window is maintained in the Java application, so there is no performance loss by transferring huge amount of video data captured by the native Kinect code to Java and display the image there (by pushing the data back to the native side by the Java Virtual Machine). Moreover the Kinect SDK examples use DirectX to boost the video performance and real-time video frame rates of more than 25 frames per second can be achieved with moderate CPUs. All other managing functions provided by the native SDK can be invoked by Java code using the simple method invocation model of JAW.

Skeleton tracking is performed by native code in the Microsoft JDK routines. The pattern recognition algorithm is known to be efficient and is improved from one JDK release to the next. With every captured video frame the tracking algorithm returns 2-dimensional positions of 20 skeletal points shown in the picture below.

20 skeletal joints and their naming convention

Whenever new data becomes available, the coordinates together with a skeleton identifier is fed into a native FIFO buffer from where Java extracts them. Because only few data is transferred between the two systems, data loss and blocking is unlikely.

A demonstration example

One of the usual demonstration examples of Kinect is a skeletal tracker. With KinectJLib and JGameGrid the program is simplified by an order of magnitude compared to the C++ code example distributed with the Kinect JDK. The code is explained in the JGameGrid web site (see).

Execute the program locally using WebStart.

System prerequisites:

• Windows 7 or higher (32- or 64-bit) Installation of the Kinect SDK from http://kinectforwindows.org
• Kinect for Xbox 360 or Kinect for Windows (both devices are supported)

(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)

Running native code under 64-bit Windows

The JAW based native code is loaded and runs as Windows DLL. KinectHandler.DLL is a Windows 32-bit DLL and KinectHandler64.DLL is its 64-bit counterpart. The loading of 32-bit and 64-bit DLLs is not optimal, neither with Windows 7 nor with Java. Under 32-bit Windows, 64-bit DLLs are not functional. On the other hand under 64-bit Windows, both 32-bit and 64-bit DLLs are supported. Because the Java Virtual Machine loads the DLL, the decision which DLL version is needed does not depend on the OS, but on the JVM. Because both JVM versions may reside on the same Windows 64-bit machine, problems while loading DLLs are frequent. To make a Java program portable between 32- and 64-bit JVMs your code must check which JVM is running and load the appropriate DLL.

Future extensions:

• Better detection of two skeletons
• Integration of speech recognition using Microsoft.Speech
• More application examples