11
\$\begingroup\$

If graphics are rendered for a large viewing angle (e.g. a very large TV or a VR headset), the viewer can't actually focus on the entire image, just a part of it. (Actually, this is the case for regular sized screens as well.)

Combined with a way to track the viewer's eyes (which is mostly viable in VR I guess), you could theoretically exploit this and render the graphics away from the viewer's focus with progressively less details and resolution, gaining performance, without losing perceived quality.

Are there any techniques for this available or under development today?

\$\endgroup\$
  • \$\begingroup\$ It's an interesting idea, but I guess it isn't viable because most displays don't come with an eye-tracker. I also think that switching the geometry at the rate of eye-movement could have a negative impact on performance. Of course you could design your level/scenes in a way that encourage the player to look at certain points of interest and provide hi-detailed meshes there while you use lower quality assets for the surroundings... \$\endgroup\$ – bummzack Jun 13 '14 at 7:05
  • 1
    \$\begingroup\$ Check out this Siggraph 2014 paper: graphics.cs.cmu.edu/projects/multirate . I haven't had time to read it, but it describes a way to shade an image at less than "1x fragment shader per pixel" execution and still get a reasonable image. Perhaps whatever algorithm or ideas they outline can be used to reduce the sampling rate on the edges of the screen and improve performance. \$\endgroup\$ – TravisG Jun 13 '14 at 15:13
  • \$\begingroup\$ Another paper that might be interesting: software.intel.com/en-us/articles/coarse-pixel-shading \$\endgroup\$ – TravisG Jun 13 '14 at 17:57
  • \$\begingroup\$ This is a thing now, called Foveated Rendering. roadtovr.com/… \$\endgroup\$ – Bart van Heukelom Jan 11 '17 at 12:19
7
\$\begingroup\$

There are examples of such effects you are talking about in many games. First, there is Depth of field, which makes object at a distance look blurred, just as if the camera is focusing on things only close to it, like in this example. You can explore more samples related to this in the DirectX SDK package- they have sample code there too. For OpenGL, you will get them on the net if you search.

enter image description here

The second thing you may want to see is what is maybe known as Geometry MipMaping or Progressive Meshes which, like the name suggests, is similar to texture mip-mapping, but for geometry. The number of polygons in a mesh is reduced as the object gets further away from the camera. This helps in increasing performance. In D3D, there exists a function which will help in generating Progressive meshes automatically. Here it is:

HRESULT WINAPI D3DXGeneratePMesh(
                                  LPD3DXMESH pMesh,
                                  const DWORD *pAdjacency,
                                 const D3DXATTRIBUTEWEIGHTS *pVertexAttributeWeights,             
                                  const FLOAT *pVertexWeights,
                                  DWORD MinValue,
                                  DWORD Options,
                                  LPD3DXPMESH *ppPMesh
                                 );

Here's an example of this:

enter image description here

EDIT: Consider the following image- enter image description here

This is the camera's culling frustum. The line that goes right through the middle is the LookAt vector. Now, assuming the player will always want to focus at the centre of the screen, you define the angle x. Any object that will be positioned at an angle more than the decided x will have a low resolution rendering, and the ones which are within the angle will be rendered with detail. Like this, the objects at the centre of the screen will be more detailed than the ones on the sides. For if the player is not looking at the centre, then just adjust that centre line to the direction in which the player is looking at. ( or maybe rotate the whole camera to that direction).

\$\endgroup\$
  • \$\begingroup\$ Although I doubt you could gain performance by implementing dof. \$\endgroup\$ – badweasel Jun 13 '14 at 6:40
  • \$\begingroup\$ @badweasel Well, that's true. DOF does not provide improvements in performance, but it is an effect worth mentioning. It Enables you to hide the low detailed, chunky geometry which just might sometimes pop out at a distance. \$\endgroup\$ – The Light Spark Jun 13 '14 at 6:49
  • \$\begingroup\$ I up-voted your answer. It's a good answer. I just wanted to point it out since it was part of his question. \$\endgroup\$ – badweasel Jun 13 '14 at 6:50
  • \$\begingroup\$ These techniques use the z distance away from the camera, but I mean the x/y distance away from the point on the screen the user is looking at. I suppose they could be adapted as such though. \$\endgroup\$ – Bart van Heukelom Jun 13 '14 at 9:58
  • 1
    \$\begingroup\$ @BartvanHeukelom Why, if the player is not focussing on the centre, you make the camera turn to make what he is fucussing at the centre. Sounds fun. \$\endgroup\$ – The Light Spark Jun 13 '14 at 15:43
2
\$\begingroup\$

Yes, Foveated Rendering utilizes eye tracking to only render what the user is looking at in high detail. Everything else, can be rendered in lower detail.

enter image description here

While it's fairly obvious when looking at a screenshot like this, it's reportedly invisible to the user in VR.

The existing mainstream headsets don't include eye tracking (yet), but there are a few after-market solutions to add eye tracking. Further a small amount of additional processing is required for the rendering process. However the benefits are significant. With foveated rendering implemented, only a small percentage FOV needs to be rendered in high detail, this can result in dramatically higher frame rates. Exact numbers depend on the implementation.

Further, this isn't even limited to VR headsets. It can be used on desktop monitors as well (with the addition of eye tracking and to a more limited effect).

\$\endgroup\$
1
\$\begingroup\$

Expanding on Byte56's answer, yes this sort of thing is being developed in the industry. Fixed Foveated Rendering is a technique employed by Valve in their robot repair demo (https://youtu.be/DdL3WC_oBO4?t=769) and in a few commercial games (Resident Evil VII for PSVR comes to mind). Fixed Foveated Rendering does not employ eye tracking but this means it works well with current versions of VR headsets which currently do not have any sort of eye-tracking capabilities. Though right now the Vive is supposed to get an eye-tracking upgrade kit so you may see some AAA VR titles utilize that.

You won't find many code-samples available related to this sort of thing but Nvidia does have some hardware extensions that are decently documented. Multi-resolution is fairly well explained but is a feature tied to their hardware.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.