1
\$\begingroup\$

In this link: https://developer.oculus.com/documentation/unity/unity-ovroverlay/

Objects and textures rendered as compositor layers also render at the frame rate of the compositor, the refresh rate of the HMD, instead of rendering at the application frame rate. Because of this, the compositor layers are less prone to judder and are ray-traced through the lenses. This improves the clarity of the textures or text displayed on them.

What does "ray-traced through the lenses" mean in this case?

\$\endgroup\$

1 Answer 1

1
\$\begingroup\$

There are two major complications when rendering content for VR:

  • Head Movement: between the time you started rendering and the time the frame becomes visible to the user, they may have turned their head slightly, causing the image to lag behind or judder.

  • Lens Distortion: to achieve a wide field of view and a comfortable focus depth, head-mounted displays use heavy-duty lenses. These have a side effect of substantially distorting the screens behind them – so where you see a pixel in space is displaced from where it's actually located on the screen.

To compensate for these effects, it's conventional to re-project the rendered image.

We start with a frame buffer containing a pair of images in linear perspective — each one similar to the image a game would output when rendering to a flat screen. This lets us use all the game engine features we ordinarily would, and take advantage of GPU rasterization hardware that makes drawing 3D content in linear perspective super fast.

Then, just before we send the new frame to the display panels inside the HMD, we re-sample it, moving and distorting the images to compensate for the lens effects and incremental head movement. The trouble is, since we're doing these operations on a raster image, the result is lossy — like scaling up an image that's too small, we can get smeared or blurry pixels in the output.

What the Oculus docs are describing here is an alternate path, where rather than rendering the image first in linear perspective and then bending/warping it, they render it so that it has the right warp to begin with.

I don't know the details of how Oculus implements this, but the basic principle would be like casting a ray from each screen pixel through the lens to see how it gets warped, then casting that adjusted ray back into your overlay geometry. Wherever it hits that overlay, that tells us the exact colour the user should see in that direction, and we send that colour to the screen pixel we started with.

Because we're figuring out one exact colour for each pixel, not blending them according to a resampling algorithm, we can potentially get crisper, more accurate VR rendering this way — especially helpful for text or content with a lot of motion.

But it's likely not as efficient as conventional rasterization and resampling, which is why it's not something we can use for all scene content, like higher-poly animated meshes or layered particle effects — at least not yet.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .