| |
The source code of all examples is included in the JGameGrid distribution.
Kinect Natural User Interface, Skeletal Tracking
(For more information consult the KinectJLib web site.)
The Kinect controller is a natural user interface (NUI) that can track persons by a depth sensing video camera and generate skeletal information for 20 body joints in real-time. It also contains a microphone array to capture sound with direction information. There are numerous potential uses including game control, robotics, surveillance, input device for exhibition stands, people tracking, etc.
In 2010 a non-official Kinect driver with a C++ API became available and is now distributed through Code Laboratories and OpenNI. There are also some Java wrappers, especially with Greenfoot, but the installation and use is pretty tricky. In 2011 Microsoft release an official development system for the Kinect device called Kinect SDK. It's fully based on the Visual Studio IDE and all examples are in C++ and C#. As usual with Microsoft, it is restricted to the Windows OS and runs only under Windows 7 and up. As soon as this official release became available we decided to create a simple Java wrapper library KinectJLib based on our Java API Wrapper (JAW). Moreover we added the class GGKinect to our game development framework JGameGrid to shield the developer completely from graphics animation issues.
One of the usual demonstration examples of Kinect is a skeletal tracker. With KinectJLib and JGameGrid the program is simplified by an order of magnitude compared to the C++ code example distributed with the Kinect JDK.
By instantiating the GGKinect class a window is displayed that shows the video image. Because the window is completely managed in native code, there is no need to transfer video data to Java. On the native side, DirectX is used to enhance the quality of the image. Even with moderate computing power, a refresh rate of more than 20 images per seconds in 640 x 480 pixel resolution is obtained. The skeletal extraction, a rather complicated pattern recognition operation, is also performed in native Kinect SDK code and only the coordinates of the 20 skeletal joints are transferred to Java using the getJoints() method. The joints are instances of class Point3D with coordinates (x,y,z). x and y correspond to the video window (left to right, 0..windowWidth, top to bottom, 0..windowHeight), z is the distance to the camera in mm, not used in this simple demonstration. Finally drawSkeleton() and drawJoints() show the skeleton in the GameGrid window using the graphics methods of the class GGBackground.
Because a KinectCloseListener is registered, the callback notifyClose() is triggered when the close button of the native window is hit.
import ch.aplu.jgamegrid.*;
import ch.aplu.kinect.*;
import java.awt.*;
import ch.aplu.ggkinect.*;
import javax.swing.JOptionPane;
public class SkeletalViewer extends GameGrid implements KinectCloseListener
{
private String dllPath =
Kinect.is64bit()? "KinectHandler64" : "KinectHandler";
public SkeletalViewer()
{
super(640, 454, 1, null, false); // GameGrid window
setPosition(640, 20);
GGBackground bg = getBg();
bg.clear(new Color(128, 255, 128));
setTitle("GameGrid Kinect - Waiting for valid skeleton...");
GGKinect kinect = new GGKinect(dllPath, "Video Frame",
0, 20, 640, 480, // Position and size
GGKinect.DecorationStyle.STANDARD);
if (!kinect.isInitialized())
{
kinect.setVisible(false);
JOptionPane.showMessageDialog(null, "Initializing of Kinect failed.");
System.exit(0);
}
kinect.addCloseListener(this);
show();
Point3D[] joints = new Point3D[20]; // Initializes 20 skeleton joints
for (int i = 0; i < 20; i++)
joints[i] = new Point3D();
kinect.getJoints(joints, 0); // Blocks undefinitely
int rc = 0; // until skeleton is valid
// Tracking loop
while (true)
{
bg.clear();
if (rc != -1) // Valid skeleton
{
setTitle("Valid Skeleton");
kinect.drawSkeleton(this, joints, 5, Color.red);
kinect.drawJoints(this, joints, 5, Color.black);
}
else
setTitle("Waiting For Valid Skeleton");
refresh();
rc = kinect.getJoints(joints, 20); // Waits maximum 200 ms
}
}
public void notifyClose()
{
System.exit(0);
}
public static void main(String args[])
{
new SkeletalViewer();
}
}
|
Execute the program locally using WebStart.
System prerequisites:
- Windows 7 or higher (32- or 64-bit)
- Installation of the Kinect SDK from http://kinectforwindows.org
- Kinect for Xbox 360 or Kinect for Windows
(both devices are supported)
(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)
Sound Detection With Kinect's Microphone Array, Transfer Video To Java
In the following example Kinect's microphone array is used to detect the sound level. The microphone acts as a standard Windows sound device and its sound direction finding capabilities are not used. The sound is digitized and and the values captured by blocks in a byte buffer of selectable length. Every time the buffer is full, the sound level (maximum value) is calculated and the callback soundLevel() of the SoundListener interface is triggered to report the level of the last block.
In the same example we show how simple it is to transfer a captured video image from the native code to Java. Just call getImage() to return a BufferedImage that can be used in Java for any image transformation and display.
import ch.aplu.jgamegrid.*;
import ch.aplu.kinect.*;
import ch.aplu.ggkinect.*;
import java.awt.*;
import java.awt.image.*;
import javax.swing.JOptionPane;
public class SoundDetector extends GameGrid
implements KinectCloseListener, GGButtonListener, SoundLevelListener
{
private GGKinect kinect;
private String dllPath =
Kinect.is64bit()? "KinectHandler64" : "KinectHandler";
private final String title = "Kinect Video Frame";
private static final int captionHeight = 19;
private final int ulx = 0; // Upper left x of native window
private final int uly = 20; // Upper left y of nativewindow
private static int width = 640; // Width of native window
private static int height = 480 + captionHeight; // Height of native window
private GGButton grabBtn = new GGButton("sprites/grab.png");
public SoundDetector()
{
super(640, 520, 1, null, false);
setSimulationPeriod(20);
setPosition(ulx + width, uly);
GGBackground bg = getBg();
bg.setPaintColor(Color.black);
bg.clear(new Color(128, 255, 128));
bg.drawRectangle(new Point(0, 0), new Point(640, 470));
addActor(grabBtn, new Location(40, 500));
grabBtn.addButtonListener(this);
if (Kinect.is64bit())
{
JOptionPane.showMessageDialog(null, "Must use 32-bit JVM.");
System.exit(0);
}
kinect = new GGKinect(dllPath,
title, // Window title
ulx, uly, // Window position
width, height, // Window size
GGKinect.DecorationStyle.STANDARD, // Decoration style
1000); // Sound buffer size
if (!kinect.isInitialized())
{
kinect.setVisible(false);
JOptionPane.showMessageDialog(null, "Initializing of Kinect failed.");
System.exit(0);
}
kinect.addCloseListener(this);
kinect.addSoundLevelListener(this);
show();
doRun();
}
public int soundLevel(int level)
{
setTitle("Sound Level = " + level);
if (level > 10)
{
Fish fish = new Fish();
addActor(fish, new Location(10, 450 - 4 * level));
return 200;
}
return 0;
}
public void notifyClose()
{
System.exit(0);
}
public void buttonClicked(GGButton button)
{
}
public void buttonReleased(GGButton button)
{
}
public void buttonPressed(GGButton button)
{
BufferedImage bi = kinect.getImage();
getBg().drawImage(bi);
}
public static void main(String args[])
{
new SoundDetector();
}
} |
The Fish class just moves the clownfishes back and forth.
import ch.aplu.jgamegrid.*;
public class Fish extends Actor
{
public Fish()
{
super("sprites/nemo.gif");
}
public void act()
{
move();
if (!isMoveValid())
turn(180);
}
} |
Execute the program locally using WebStart. Make same noise to create new actors.
System prerequisites:
- Windows 7 or higher (32- or 64-bit)
- Installation of the Kinect SDK from http://kinectforwindows.org
- Kinect for Xbox 360 or Kinect for Windows
(both devices are supported)
(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)
| |
