How to achieve hologram projection effect with head tracking

I want to recreate this: http://www.youtube.com/watch?v=9xMSGmjOZIg&feature=related

I can take care of tracking the user's head, and in one of the comments I believe the author says that he used a vertex shader to do the work, but what logic do you use to move the vertices to get this effect?

• The math behind this is probably very similar to the one done with a Wii Remote + LED goggles: johnnylee.net/projects/wii (at the bottom) Commented Nov 29, 2011 at 1:27
• Any idea what that math is? Commented Nov 29, 2011 at 2:45
• If you download the sample application the source code is included. Commented Nov 29, 2011 at 6:33
• @Tetrad the Wii one simply changes the frustrum, this one looks like he is performing a distortion similar to the one that those sidewalk 3D artists do. Commented Nov 29, 2011 at 7:25
• This thesis paper: t2i.se/pub/papers/Thesis_1_Final.pdf describes part of this on page 40. How is the nearPlane value determined and scale factor(if any) do i need to apply the kinect's head position units? Commented Nov 29, 2011 at 15:01

To implement this effect, there are a few things you need to know:

1. The exact physical measurements of the display surface.
2. The viewer's eye location, relative to the centre of the screen.

Once you have this information, it's relatively simple to get the effect. Instead of calculating a normal projection matrix (typically by taking the camera position, vertical field of view, aspect ratio, and near and far clip distances), you instead construct your own skewed projection matrix using explicit frustum planes.

To generate these explicit frustum planes, you use the centre of the screen as your "zero" position, and define each frustum plane by the viewer's eye location (relative to the zero position) and two of the corners of the screen (also relative to the zero position). Near and far clip planes may either be parallel to the screen (which is a little easier mathematically) or perpendicular to the eye->zero vector.

I seem to recall reading that D3D has an explicit API for creating skewed projection matrices, while you need to do the math yourself in OpenGL. Either way, there's plenty of sample code for constructing skewed projection matrices from arbitrary frustum planes in Google.

Once you have your skewed projection matrix, all the rest of your rendering happens normally, exactly as it does with a non-skewed projection matrix. Note: do not attempt to rotate or translate a "camera" object to mimic the viewer's motion; that's all handled explicitly by the skewed projection matrix. (If you wish the viewer to be able to navigate through the world to a greater extent than they can by moving physically, say by using a joystick, then you'll need to use a camera. Remember that any such camera motion/rotation would be relative to the physical screen, not relative to the viewer's perspective, so you'd need to do some extra maths if you wanted to make the camera move relative to the viewer's facing)

Anyhow, the net effect is that the game renders a carefully distorted view of the scene, precisely calculated to not appear distorted when viewed specifically from the viewer's eye position (and nowhere else).

A few notes on the video linked in the question. First, it looks like the effect being shown in the video has the helicopter placed slightly forward of the screen. Objects forward of the screen are always tricky to do well in this sort of system, as the view frustum is so narrow that it's so easy for them to wind up falling outside of the frustum when viewed from a sharp angle. For this reason, it's much more standard to place objects behind the screen, treating the screen as if it was a limited "window" into another 3D scene. If you're going to put objects in front of the screen (or intersecting the screen), then it's important to make these objects small relative to the size of the screen, and not to place them very far forward of the screen, or you risk users walking past them, or otherwise having difficulty finding them.

Also note that in this sort of rendering approach, the object will not appear to 'pop out'; it distinctly remains a 2D image on a flat screen. It's kind of disconcerting the first time you see something doing this in real life, rather than in a YouTube video; it's not at all what most people seem to have been expecting, despite intellectually knowing that the image is still just an image on a flat screen.

Finally, note that display lag is a serious issue with this sort of effect. Your whole architecture (viewer eye detection->rendering->display) needs to be as responsive as possible in an effort to minimise motion sickness among people using the system. This is a problem suffered by every VR-like system which tries to feed people images based upon their movements; many (but not all!) people simply can't cope with more than a few milliseconds of lag, and soon begin to feel nauseous. The longer the lag, the more people seem to be affected. So keep this in mind, too.

It looks to me like you just set the camera position to the user's head position, then run the vertex shader normally.

You'd probably need to feed it through a matrix to convert whatever coordinates Kinect uses into the world space of your scene, but that's about it.

• I dont think its that simple. If you were to move the camera positions to match the head's position the object on the wall would be projected larger and larger as you moved toward it. Commented Nov 29, 2011 at 2:43
• That, and change the view frustum so that it matches the physical display in relation to the head.. potentially skewing the frustum when head moves side-to-side... Doesn't seem that easy (-8 Commented Nov 29, 2011 at 4:30