| |
The source code of all examples is included in the KinectJLib distribution.
Example 2: Kinect gestures to control a robot
In typical gaming application, the Kinect device is used to detect gestures for controlling a game object. You may even steer a real object like a car or a robot. In this example we drive the Lego NXT robot in direct mode by using left and right hand gestures. The commands are transferred to the robot via Bluetooth (see the NxtJLib website for more information).
The following gestures are implemented:
- Left or right hand up: STOP
- Left and right hands up: MOVE_BACKWARD
- Left hand down-left: MOVE_LEFT
- Right hand down-right: MOVE_RIGHT
- Left and right hand down-middle: MOVE_FORWARD
Execute the program locally using WebStart.
System prerequisites:
- Windows 7 or higher (32- or 64-bit)
- Installation of the Kinect SDK from http://kinectforwindows.org
- Kinect for Xbox 360 or Kinect for Windows
(both devices are supported)
- Lego NXT robot, Bluetooth enabled
- Bluetooth link PC-NXT
(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)
If you don't have a Lego NXT at hand or for training purposes, you can drive a simulated robot using the NxtSim library that is based on the JGameGrid framework. We add a touch sensor that detects if the robot touches the channel border. When the border is hit, the game is over. It may be restarted by rasing a hand.
import ch.aplu.kinect.*;
import java.awt.*;
import ch.aplu.jaw.*;
import ch.aplu.nxtsim.*;
import ch.aplu.jgamegrid.*;
import javax.swing.JOptionPane;
public class KinectNxtSim implements TouchListener
{
enum State
{
TURN_LEFT, TURN_RIGHT, MOVE_FORWARD, MOVE_BACKWARD, STOP
};
private String dllPath =
Kinect.is64bit() ? "KinectHandler64" : "KinectHandler";
private NxtRobot robot;
private Gear gear;
private GameGrid gg;
private boolean isOver = false;
private State state = State.STOP;
public KinectNxtSim()
{
robot = new NxtRobot();
gg = robot.getGameGrid();
gg.setPosition(10, 40);
gg.addStatusBar(40);
gear = new Gear();
robot.addPart(gear);
gear.setSpeed(50);
TouchSensor ts = new TouchSensor(SensorPort.S3);
robot.addPart(ts);
ts.addTouchListener(this);
Kinect kinect = new Kinect(dllPath, "Policeman", 550, 40, 640 / 4, 480 / 4,
NativeHandler.WS_BORDER | NativeHandler.WS_VISIBLE);
if (!kinect.isInitialized())
{
kinect.setVisible(false);
JOptionPane.showMessageDialog(null, "Initializing of Kinect failed.");
System.exit(0);
}
kinect.setWindowScaleFactor(4);
Point3D[] joints = new Point3D[20]; // 20 skeleton joints
for (int i = 0; i < 20; i++)
joints[i] = new Point3D();
gg.setStatusText("Waiting for valid skeleton...");
kinect.getJoints(joints, 0); // Blocks undefinitely
int rc = 0;
String msg;
while (true)
{
if (rc != -1) // Valid
{
Point3D leftHand = joints[SkeletonJoint.HAND_LEFT.ordinal()];
Point3D rightHand = joints[SkeletonJoint.HAND_RIGHT.ordinal()];
if (isOver && (rightHand.y < 100 || leftHand.y < 100)) // one hand up
{
robot.reset();
isOver = false;
continue;
}
if (rightHand.y < 100 && leftHand.y < 100) // both hands up
state = State.MOVE_BACKWARD;
else if (rightHand.y < 100 || leftHand.y < 100) // one hand up
state = State.STOP;
else if (rightHand.x < 320) // Right hand points to left
state = State.TURN_LEFT;
else if (leftHand.x > 320) // Left hand points to right
state = State.TURN_RIGHT;
else // both hands in neutral position
state = State.MOVE_FORWARD;
msg = "(xLeft, xRight, yLeft, yRight) = ("
+ leftHand.x + ", " + rightHand.x + ", "
+ leftHand.y + ", " + rightHand.y + ")";
msg += "\nStatus: " + state
+ (isOver ? " (Border hit. Game Over. Raise a hand to restart.)" : "");
gg.setStatusText(msg);
}
else
{
gg.setStatusText("Waiting for valid skeleton...");
state = State.STOP;
gear.stop();
robot.reset();
}
if (!isOver)
setGear();
rc = kinect.getJoints(joints, 20); // Blocks maximum 300 ms
}
}
private void setGear()
{
switch (state)
{
case MOVE_FORWARD:
gear.forward();
break;
case MOVE_BACKWARD:
gear.backward();
break;
case TURN_LEFT:
gear.leftArc(0.2);
break;
case TURN_RIGHT:
gear.rightArc(0.2);
break;
case STOP:
gear.stop();
break;
}
}
public void pressed(SensorPort port)
{
isOver = true;
state = State.STOP;
setGear();
}
public void released(SensorPort port)
{
}
// ------------------ Environment --------------------------
static
{
NxtContext.setLocation(10, 10);
NxtContext.setStartDirection(0);
NxtContext.setStartPosition(100, 240);
NxtContext.useObstacle(NxtContext.channel);
}
public static void main(String args[])
{
new KinectNxtSim();
}
} |
Execute the program locally using WebStart.
System prerequisites:
- Windows 7 or higher (32- or 64-bit)
- Installation of the Kinect SDK from http://kinectforwindows.org
- Kinect for Xbox 360 or Kinect for Windows
(both devices are supported)
- (No real Lego robot needed)
(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)
| |
Java exemplarisch
Programming Concepts in Python
LinkUp
TCPCom
JGameGrid 
JCardGame
JDroidLib 
JTurtleLib 
TcpJLib
KinectJLib
JAW 
Python/Jython
Java Utilities 
Bluetooth Lib
NxtJLib
XboxController
Simulations 
Overtone 
FtController 
