Rendering only a part of the screen in high detail

Question

If graphics are rendered for a large viewing angle (e.g. a very large TV or a VR headset), the viewer can't actually focus on the entire image, just a part of it. (Actually, this is the case for regular sized screens as well.)

Combined with a way to track the viewer's eyes (which is mostly viable in VR I guess), you could theoretically exploit this and render the graphics away from the viewer's focus with progressively less details and resolution, gaining performance, without losing perceived quality.

Are there any techniques for this available or under development today?

Answer 1

There are examples of such effects you are talking about in many games. First, there is Depth of field, which makes object at a distance look blurred, just as if the camera is focusing on things only close to it, like in this example. You can explore more samples related to this in the DirectX SDK package- they have sample code there too. For OpenGL, you will get them on the net if you search.

The second thing you may want to see is what is maybe known as Geometry MipMaping or Progressive Meshes which, like the name suggests, is similar to texture mip-mapping, but for geometry. The number of polygons in a mesh is reduced as the object gets further away from the camera. This helps in increasing performance. In D3D, there exists a function which will help in generating Progressive meshes automatically. Here it is:

HRESULT WINAPI D3DXGeneratePMesh(
                                  LPD3DXMESH pMesh,
                                  const DWORD *pAdjacency,
                                 const D3DXATTRIBUTEWEIGHTS *pVertexAttributeWeights,             
                                  const FLOAT *pVertexWeights,
                                  DWORD MinValue,
                                  DWORD Options,
                                  LPD3DXPMESH *ppPMesh
                                 );

Here's an example of this:

EDIT: Consider the following image-

This is the camera's culling frustum. The line that goes right through the middle is the LookAt vector. Now, assuming the player will always want to focus at the centre of the screen, you define the angle x. Any object that will be positioned at an angle more than the decided x will have a low resolution rendering, and the ones which are within the angle will be rendered with detail. Like this, the objects at the centre of the screen will be more detailed than the ones on the sides. For if the player is not looking at the centre, then just adjust that centre line to the direction in which the player is looking at. ( or maybe rotate the whole camera to that direction).

Answer 2

Yes, Foveated Rendering utilizes eye tracking to only render what the user is looking at in high detail. Everything else, can be rendered in lower detail.

While it's fairly obvious when looking at a screenshot like this, it's reportedly invisible to the user in VR.

The existing mainstream headsets don't include eye tracking (yet), but there are a few after-market solutions to add eye tracking. Further a small amount of additional processing is required for the rendering process. However the benefits are significant. With foveated rendering implemented, only a small percentage FOV needs to be rendered in high detail, this can result in dramatically higher frame rates. Exact numbers depend on the implementation.

Further, this isn't even limited to VR headsets. It can be used on desktop monitors as well (with the addition of eye tracking and to a more limited effect).

Answer 3

Expanding on Byte56's answer, yes this sort of thing is being developed in the industry. Fixed Foveated Rendering is a technique employed by Valve in their robot repair demo (https://youtu.be/DdL3WC_oBO4?t=769) and in a few commercial games (Resident Evil VII for PSVR comes to mind). Fixed Foveated Rendering does not employ eye tracking but this means it works well with current versions of VR headsets which currently do not have any sort of eye-tracking capabilities. Though right now the Vive is supposed to get an eye-tracking upgrade kit so you may see some AAA VR titles utilize that.

You won't find many code-samples available related to this sort of thing but Nvidia does have some hardware extensions that are decently documented. Multi-resolution is fairly well explained but is a feature tied to their hardware.