Access various projections and key matrices

[ahundt]

There are a few different functions that would be nice to have.

• color_to_depth
• Various transformation matrices
  • IR image x,y to Color x,y and back
  • IR cloud xyz to color x,y and back
• Getting the intrinsic matrix
  • I know there is Freenect2Device::ColorCameraParams, but for some reason once I put it into a matrix and called cv::decomposeHomographyMat(H, K, rotations, translations, normals); from OpenCV, none of the solutions have been projecting correctly for me.
  • There is that interesting predefined constant... does that matter in that case?

[xlz]

Previous discussion: #223 #41

There are some problem extracting these matrices losslessly from the builtin parameters.

[ahundt]

Here are example motivating use cases that can be used to help determine what is actually needed:

• Known object detection and pose estimation
• narrow down the search space via the color image
  • color_to_depth would be useful here
• project point cloud models of objects into the frame of the kinect
  • As derived from Homography between two images (example video)
  • Projecting the object model into the frame (video of broken first attempt that illustrates the idea well)

@xlz those previous discussions are definitely relevant, and alternative ways to achieve this aside from the standard transformation matrices are definitely worth considering, particularly if they are more accurate.

Haven't thought about it yet but it is probably also worth determining if color_to_depth can be inverted in a reasonable manner.

[floe]

My knowledge of multiple-view geometry is a bit rusty, but IIRC, for the special case of the Kinect (one color camera & one depth camera), the exact transformation from one camera space to the other always depends on the live depth values and therefore can't be expressed as a "classic" transformation matrix, right?

Did you already have a look at the bigdepth parameter to Registration::apply? (https://openkinect.github.io/libfreenect2/classlibfreenect2_1_1Registration.html#a8be3a548ff440e30f5e89a4b053e04fa) This may be useful to address at least some of your issues.

[ahundt]

• bigdepth image side black borders #509, Color pixel to depth coordinates #419 are also potentially relevant issues
• calibration libraries by arpg might provide some interesting ideas as well.
  • Calibu library
  • Calibration docs
  • their hardware abstraction for libfreenect2 interface

[ahundt]

@floe Yeah, it seems there is a bit more to it because the depth emitter and receiver need to both be accounted for in addition to the rgb sensor.

It looks like there are some potentially useful resources (papers, etc) in the following google results, and these results.

Some of those are for the original Kinect, so of course structured light vs ToF differences apply where appropriate.

[floe]

You're right, my previous post was slightly incorrect. From one camera space to the other, you can indeed use undistortion + regular transformation matrix. However, to actually map one pixel to the corresponding one in the other image (i.e. from one image space to the other), you need the live depth values.

[ahundt]

Sorry, my edits made this confusing. For future readers, I originally made a note of the similarity between camera field of views in stereo vs depth + rgb cameras. I then realized there is more complexity to this than the stereo case and edited my post.

[xlz]

Let me describe the current status:

• The intrinsic parameters of the depth camera are known from the factory builtin parameters.
• The transformation of a 3-D pixel from depth to color camera is known from a builtin formula (kinect2_depth_to_color), but it is not in a matrix multiplication form.
• The extrinsic matrix between the depth and color camera is not known.
• The Some intrinsic parameters (distortion terms) of the color camera are not known. (Focal lengths, principal points are known.)

I gave it a go #223 (comment) trying to figure the extrinsic matrix from kinect2_depth_to_color but did not succeed. You can always look at it again.

To use matrix for extrinsic transformation requires either you supply the matrix by external means, or you figure out the extrinsic matrix from kinect2_depth_to_color.

[floe]

To use matrix for extrinsic transformation requires either you supply the matrix by external means, or you figure out the extrinsic matrix from kinect2_depth_to_color.

IIRC kinect2_depth_to_color also subsumes the intrinsic parameters of the color camera?

[xlz]

I think yes, in #223 there is a plot showing that there is some (tiny) distortion produced by kinect2_depth_to_color.

[xlz]

By the way, I found one old proposal:

I'd say using numeric methods to solve kinect2_color_to_depth() would be sufficient.