Kinect 2.0 Joint / JointOrientation to Ogre3D

[Elenthidir]

Hey guys,
I want to reimplement the Demo of Sinbad using Kinect v2, but I am encountering a problem which I was not able to solve on my own...

The connection to the kinect is working and I can print some values like the joint position (x,y,z) and the joint orientation(x,y,z,w) for each of the 25 joints found by the Kinect skeleton.
My problem is now how I can map those values to sinbad? I know there is a demo for the kinect v1 using OpenNI, but I can't find the sourcecode.

In this posting it says that the orientation is "roll", "pitch", "yaw". Some people say that the values are absolute and other that they are relative to the parents bone.

But my problem is, that I do not know how to create a quaterion from r,p,y... If I have the quaternion I can apply it to a bone, that wouldn't be the problem.

Here are some values coming from the kinect when moving the left arm. (Values are for the wrist) first the 3 position values and after that the 4 orientation values, each line for one frame of the kinect

Code: Select all

11:50:39: WRIST: 0.105007 -0.0241627 0.890557 -0.0275453 -0.197223 0.976617 -0.0810145 
11:50:39: WRIST: 0.168085 -0.0717494 0.920979 -0.0700791 -0.0294791 0.992532 -0.0953983 
11:50:39: WRIST: 0.245335 -0.0582655 0.831103 -0.126986 -0.189362 0.950926 -0.20918 
11:50:39: WRIST: 0.261235 -0.0738448 0.889838 -0.281718 -0.0110119 0.928249 -0.242625 
11:50:39: WRIST: 0.386774 0.038957 0.855162 -0.4302 -0.316903 0.661656 -0.526034 
11:50:39: WRIST: 0.381307 0.0654915 0.877569 0.509834 0.250843 -0.554102 0.608373 
11:50:39: WRIST: 0.365752 0.0196074 0.864313 -0.392745 -0.0836547 0.703819 -0.585996 
11:50:39: WRIST: 0.358447 0.0287535 0.85712 -0.264221 0.0757864 0.713433 -0.64456 
11:50:39: WRIST: 0.366006 -0.122044 0.561175 -0.160404 -0.344358 0.872678 -0.306792 
11:50:39: WRIST: 0.37519 -0.11875 0.550461 0.0667296 -0.340323 0.858438 -0.377905 
11:50:39: WRIST: 0.394419 -0.117494 0.536459 0.265227 -0.22559 0.805716 -0.479151 
11:50:39: WRIST: 0.437061 -0.104316 0.543979 0.156031 -0.13026 0.820895 -0.533684 
11:50:39: WRIST: 0.460237 -0.091752 0.567488 0.00790161 -0.111821 0.855681 -0.505216 
11:50:39: WRIST: 0.437995 -0.104449 0.581361 0.930455 0.363881 -0.0069316 -0.0423697 

Can someone help me and at least explain to me what those values are or how I can use roll pitch yaw to create a valid quaternion?

[Kojack]

Those already look like valid quaternions. The first three numbers are the position vector, the next 4 numbers are the quaternion. There's no pitch/yaw/roll there that I know of.

If you do want to get euler angles into a quaternion, the simplest way is via a Matrix3.
Matrix3 has 6 methods called FromEulerAngles (with different combos of X, Y and Z on the end). These convert euler to a Matrix3. Then you can give the matrix to the quaternion constructor, or call the quaternion's FromRotationMatrix method.

I don't have the kinect2 sdk with me (at work) but from what I just saw on the net, the joint orientations are absolute quaternions, which will rotate a coordinate system so the Y axis points along the bone. It doesn't appear to be a hierarchy (that was in the old sdk). Ogre uses a hierarchy of bones, so some work will need to be done to get a flat group of absolute bones into ogre's relative hierarchy.

[Elenthidir]

Thanks for your answer.
The values are quaternions where the y orientation follows the direction to its parent. This is in CameraSpaceCoordinates.
Can someone help me how I can apply those values to the bones? Currently I am using this, not working, approach.

Code: Select all

Ogre::Bone* bone = sinbad->getSkeleton()->getBone("Humerus.L");
bone->setManuallyControlled(true);
bone->reset();		
bone->rotate(skeleton->getBoneByName(KinectStruct::BONES::HUMERUS_L).orientation, Ogre::Node::TS_PARENT);

The arm of sinbad is doing some strange things but not following the movement of my arm in front of the kinect in any way...

The skeleton-class contains the quaternions from kinect without any further processing.

Humerus_L:

Code: Select all

	KinectStruct::BONE humerusL;
	humerusL.orientation = shoulderL_ori;
	humerusL.position = shoulderL_pos;

I am using the shoulder as written here:http://www.ogre3d.org/forums/viewtopic.php?f=2&t=69938 written in the first answer

Here is a YT Video which shows how it currently fails: https://www.youtube.com/watch?v=QC8SnFBb5sg

[Elenthidir]

I've solved my problem:

I've tried to apply the orientation of the elbow to the ulna but the data from the kinect is as following.
If you want to have the orientation of the humerus for example you need to get the values from the elbow.
Reading a lot of posts in the official MS forum, they always said that I have to multiply the kinect joint orientatin with the inverse of the parent, and than I have the relative orientation of the current bone.

This fails badly, because I did a huge misstake... I've only used the kinect data and totally ignored the OgreData. I've now used the bones parent for my calculations and it works perfectly.

Here is my code (sorry for the bad formatting, this is my testcode. the final code will be much better):

Code: Select all

	Ogre::Bone* bone = sinbad->getSkeleton()->getBone("Humerus.L");
	bone->setManuallyControlled(true);
	bone->resetOrientation();	
	Ogre::Quaternion parentOrientation = bone->getParent()->_getFullTransform().extractQuaternion();

	Ogre::Quaternion shoulder = KinectHelper::kinectOrientationToOgreQuaternion(&skeleton->getKinectJointOrientation(JointType_ElbowRight)->Orientation);

	KinectHelper::convertCoordinateSystem(&shoulder);

	Ogre::Quaternion orientation = parentOrientation.Inverse() * shoulder;	
	bone->setOrientation(orientation);

With this code, the Left arm of Sindbad is following my arm in the kinect.

One little bit: You need to take the data from the RIGHT arm for the LEFT arm of sinbad. the convertCoordinateSystem Methis will make the left handed system form the kinect to a right handed Ogre System.

Hope I can help someone with this code, which is working for me.