1
\$\begingroup\$

When reading about the usual methods of rendering 3D models in game programming one encounters ideas such as the Perspective Projection Matrix (rasterization), Ray tracing and Ray marching. On the other hand, modern physics brought about a holographic conjecture about the universe, which would mean that it is possible to completely describe phenomena in three dimensions by using only two dimensions, in a certain sense being able to describe the movements on a volume of space by the conditions on its boundary, analogous to what happens in the holographic film of a hologram.

Knowing this, one cannot help but wonder if there was ever any implementation of this holographic principle in 3d graphics rendering. The Perspective Projection Matrix method of rendering seems to be the least computationally demanding method, but it involves calculations with a 4x4 matrix, where as the holographic principle would suggest that one might be able to use a 3x3 Holographic Perspective Projection matrix on two dimensional data that is encoded in this holographic form, wouldn't this increase computation performances?

\$\endgroup\$
1
  • \$\begingroup\$ Please ask one question per post. I've removed the second question about other alternative 3D rendering methods. The old Wolfenstein-style rendering using mostly 2D you mention is called Raycasting. You may want to check out past Q&A like "Are there viable alternatives to polygons for creating 3D worlds?" and then post a new question if there are still aspects beyond this that you're curious about. \$\endgroup\$
    – DMGregory
    Commented Aug 29 at 12:54

1 Answer 1

2
\$\begingroup\$

It's a neat idea, but unfortunately there are a few problems that make it not a practical approach to games (at least for now).

  1. We don't actually know how to simulate a 3D universe like ours on a 2D boundary.

    It's been shown to work for other types of universe with different curvature / dimensionality than ours, and so in principle it seems there should be a way to formulate the laws of our familiar 3D universe on a 2D surface, but it's still an active area of research that physicists are working on. We don't have a ready-to-go set of equations we can just drop into a game engine.

    But even if we did...

  2. It's not the kind of model we need.

    Holographic models are typically extensions of quantum field theory, representing action in the universe as waves rippling through fields of force and complex probability amplitudes. It's a very powerful and elegant view of the universe, with amazing emergent properties like this holographic trait.

    ...but it's not clear how to use them to answer the questions we ask our game physics engines like "is this cube intersecting this sphere?" or "did my bullet hit?" — at least, not without simulating the wave functions of every quark, gluon, electron, etc. making them up. It's a very reductive way to model physics, and even our best supercomputers and cleverest approximation models can only simulate a handful of particles from first principles like this. Game simulations typically work at a vastly higher level of abstraction, chunking all that messy quantum weirdness up to the level of the rigid body dynamics that emerge from the raw physics in an approximate statistical limit.

    But hey, maybe you're making a game about manipulating just a few subatomic particles. Even then...

  3. Projecting the hologram isn't "easy".

    The full math of how these holographic models work is beyond the scope of an answer here, and beyond my level of math and physics understanding, but we can get a sense of it from a simplified model presented in this PBS Space Time video.

    Fractal cellular automaton animation

    We can imagine the physics of an idealized universe as a cellular automaton running on an immense 2D grid. Patterns of cells interact at the smallest scales, and together form larger patterns that interact on larger scales, a bit like unit cells simulating "Life in Life", nesting fractally to larger and larger scales. One way to project such a 2D universe to a 3D hologram is to take the patterns at each scale factor and put them on a 3D shell, nested like a matryoshka doll. You end up with patterns moving around in 3D space, even though the underlying simulation rule is 2D.

    Nested shells

    This works, though it's not obvious how to go directly from the 2D simulation grid to a 2D rendering of a viewpoint in the holographic 3D space implied by that grid without first extracting a neighbourhood of that 3D hologram as a middle step. But it gets worse...

    Notice that in order to evaluate what's happening in a patch of 3D space (say, the area around the player), we now need information that's spread out across a large area of the underlying 2D simulation. Taking just a tiny step "inward" toward a coarser-grained shell means we need to evaluate patterns on a larger scale of the grid, processing many many more cells that are spread farther apart.

    This kind of non-local calculation happens "for free" when your computer is the wavefunction of our universe's physics itself. But it's somewhat antagonistic to the way our matter-based computers are built. CPUs, GPUs, and memory/cache systems like having "data locality", where all the inputs they need to compute an output are located close together. But with the hologram, we need to keep track of interactions at all scales and separations.

    This is computationally challenging, especially because...

  4. Doing calculations on 3D vectors isn't that hard. (And likewise for 4D homogeneous vectors)

    Modern CPUs and GPUs contain specialized circuitry called SIMD or vector units that can process groups of data in parallel. So adding two 4D vectors isn't necessarily twice the work as adding two 2D vectors - it's actually the same number of instructions and computation cycles either way. (There are throughput/bandwidth/caching benefits to having fewer data members, but as we saw above, non-locality trashes that anyway). Because 4x4 matrix computation has been so heavily used in computer graphics for half a century, dedicated hardware to make it fast is commonplace.

    That means...

  5. Modern games are not usually bottlenecked on simulation or projection.

    What kills us these days is "fill rate". As high-resolution 4K, 5K+ displays have gotten more popular, we have a huge number of pixels that need new colour values computed each frame. Multiply that by everything happening at each pixel: overdraw, lighting, layers of transparency, post effects... so it's usually these colour calculations and the memory bandwidth of squeezing all those results out to the frame buffer that's the limiting factor. That's why upscaling and frame interpolation techniques are so popular now.

    The CPU that (usually) runs the physics simulation, and the GPU vertex shaders that handle the projection matrix math are rarely the slow parts holding us back - especially when it comes to purpose-built gaming PCs or consoles. You might get bottlenecked here on mobile platforms, but those are probably the last place you want to try to run a full holographic simulation of physics. 😅

So, for the moment at least, it's a neat idea but not a practical solution. Who knows, maybe future developments in physics theory, software algorithms, and hardware technology will turn this into a practical shortcut for game simulation and rendering workloads. From where we are just now though, it's difficult to even speculate on what kinds of advancements would be needed for that to happen.

\$\endgroup\$
1
  • \$\begingroup\$ Minor note: real-time raytracing can hit the CPU pretty hard. A system with an older CPU but newer raytrace-capable GPU can easily bottleneck on the CPU side when raytracing. \$\endgroup\$
    – Kevin
    Commented Aug 29 at 16:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .