've just started building a physical camera class for the depth data you get from a Kinect. I have a Vector3 array of [imageWidth, imageHeight] describing the LOCAL positions I get in from the kinect (once I project them back at the FOV of the camera lens etc). I then translate, scale and rotate the whole virtual kinect camera in space.
What I'm looking to do is to extract back again the positions of the kinect camera input but in WORLD space, not LOCAL space (i.e. it's gone through the translation, scale and rotation and gives me a Vector3 for each value of the kinect in WORLD space).