3D-lighting considerations

Question

I'm considering how a graphics-engine lights any given environment, at the level of code.

So I guess perhaps the simplest way to light a scene is just to provide a Value-figure (i.e. Value as in the HSV colour-model) to each pixel according to whether you think the area of the scene is more or less in shadow, and just forget about light direction and light sources etc. I guess this is probably how it was done until relatively recently, and probably is still done this way in a lot, if not most, cases. This way being very lightweight in terms of calculations and processing.

A slightly more demanding approach for an outdoors scene in which day-night are operational would be to again assign a Value-figure to each pixel, but also give each a ‘sun-height’ value between 0-180, where each coordinate's Value-figure maximises as that ‘sun-height’ value reaches 90, and minimises at 0 and 180. The rate at which that ‘sun-height’ value changes simulating the course of a day. And then to avoid pitch blackness, keep each coordinate's Value-figure above-zero at its very minimum, so as to provide each with enough plausible visibility in the absence of the sun.

A more demanding approach further still is to then have each coordinate's Value-figure increase in accord with its relative position to a point light-source. So if indoors, and there's a single medieval torch on the wall, then let the light-source of the room be the coordinate that is exactly at the centre of that torch; and then let each coordinate's Value-figure increase in-proportion to its relative angle to, and distance away from, that point. Again, give each pixel a minimum Value-figure below which it can't drop (as before) in the absence of the torch. This now becomes more difficult as secondary-reflections have to be considered a little bit, which I'll say a little bit about now.

Regarding secondary-reflections, in theory I guess you let each coordinate that is illuminated by the light-source become its own light-source of a lesser intensity to the original's, which then illuminates further any pixels within its range, and which pattern then continues until the reflected light's intensity drops to zero. But in practice this must surely be impossible, or at the very least extremely impractical to execute at a reasonable frame-rate and hindering to the game's functioning. So I guess secondary-reflections should be limited to simply looking at the environment and deducing which areas would have more or less reflected light enter them and then scale the Value-figure of coordinates in those areas accordingly.

The most demanding approach--excepting the extreme case of millions of secondary-reflection calculations that I just talked about--would be to calculate for a point-light-source as before, but to do so for multiple light-sources: so if there were 6 torches in a corridor as well as a fire-place, then calculate each coordinate's Value-figure according to the angles it tends with, and its distance from, each of those torches as well as the fire-place.

Secondary-reflection for the side of an object that faces away from the light-source would be challenging without doing those extreme calculations I mentioned earlier: how otherwise do you judge how dark the away-facing side of a sphere should be whose opposite side faces a torch? Surely you can't just let the Value-figure of any coordinate that has a different coordinate ‘eclipsing’ it be zero, as there has to be some gradient…

Answer 1

I'm not a game dev, but I did take a class on this in college, and can give a brief tldr.

1990-1995 - CPU Rendering

The oldest "3d" games straight up did not take lighting into account, as they were rendered on the CPU, which lacked the processing power to do much per-pixel . The developers just added the shadows to the objects textures directly. Computers generally ran at a resolution around 640 × 480, and if you had a target framework 24fps, and wanted to set each pixel separately, then they had to run in a loop, which was 22,118,400 operations per second. Computers at the time could only do about 8,000,000 operations per second, so even if just setting all pixels to hardcoded white one at a time wasn't viable at that resolution and framerate.

Games used insane tricks to calculate 3d-like graphics at very low framerates and very low resolutions. One of the most fastest of these was made by John Carmack, "Wolfenstein 3d" ran at 320x200 at roughly 20fps. Wolfenstein3d (1992) pulled this off by only having 3d walls, straight up and down. If you look closely, it doesn't actually even have floors or ceilings. Everything else was 2d sprites. In 1993, John Carmack made an even better "Doom" engine in 1993, which was able to add sloped floors and ceilings, but walls were still perfectly vertical. Each wall/ceiling/floor could have hardcoded light+color set as part of the level.

1995 - 2007 - Fixed Pipeline GPUs

In 1995/1996, the Nintendo 64, PlayStation, and Sega Saturn released, with dedicated "GPUs", and shortly therafter, the first consumer personal computer dedicated 3D graphics cards became available. These generally work in a relatively straightforward graphics pipeline. First, The CPU sends commands to the GPU telling it what to add to the next frame. These usually include:

• where the camera is (as a transform matrix, handling camera location, size, rotations, and other weird scaling things)
• Ambient light data (color + intensity).
• objects:
  • The address of a 3d model
    • vertex (corner) coordinates and "normal vectors".
    • faces (trianges) connecting these vertexes.
  • The address of the textures to use for the model.
  • A transform matrix (This handles location, size, rotations, and other weird scaling things).
  • Directional light source data (direction + color + intensity).
  • other data.

Then the pipeline runs:

1. Projection: For each "vertex" (corner) in each model, the GPU multiplies the 3d coordinates by the model transform matrix, to figure out where this vertex actually is "in the space", and the camera transform matrix, to figure out where this vertex actually is relative to the camera. (GPUs actually merge this into a single step).
2. Lighting: For each "vertex" (corner) in each model, The GPU looks up the "normal" vector (aka "what direction is this point on the object facing"), and calculates the lighting for that vertex:
   • "ambient light": the light that bounces everywhere, setting a minimum lighting in a space. This is usually just passed in, not calculated.
   • "directional light": The light from the light source to use. Usually the sun, but sometimes toward a nearby light source. The more the point faces the light source, the more of this light it "absorbs". If it faces perpendicular or away, then it does not absorb any directional light.
   • "camera light": Weirdly, things look better if we calculate as if the camera itself were projecting a light. The more the point faces the camera, the more of this color it "absorbs". If it faces perpendicular or away, then it does not absorb any camera light.
3. Clipping: For each "face" (triangle) in each model, the GPU discards any triangles outside of the camera, or facing away from the camera.
4. Window-Viewport transformation: For each "vertex" (corner) in each model, a final matrix transformation is calculated, converting the relative camera coordinates to 3d pixel coordinates.
5. Rasterization: For each "face" (triangle) in each model:
   • the GPU looks up the pixel coordinates and lighting of its vertexes, and the texture for the face.
   • For each pixel in that face:
     • The GPU checks if a color has already been chosen for this pixel that is closer to the camera. if so, it skips this pixel.
     • The GPU looks up the texture color for that point in the triangle.
     • The GPU interpolates the lighting for that point based on how close it is to each of the three corners.
     • The GPU stores the pixel color, including how far it is from the camera.

2000-2007 Shaders

Around 2000, GPUs started releasing where the Rasterization step wasn't directly wired, but instead ran the Rasterization step as a program (aka "Pixel Shader") on GPU processors. This also allowed applications to replace the program, to do customized shading of the pixel. This could be used to calculate textures at runtime, reducing code size and memory usage, and making the textures dynamic. Several games used this to make realistic pupils that always watched the camera/player.

Almost immediately after, GPUs also did the same for the Projection step, allowing applications to replace the Projection with a custom "Vertex shader" . This enabled programs to make models "morph", allowing super smooth 3d animation, rather than simply rendering.

Around 2007, GPUs also added "Graphics Shaders" (aka "Tesselation Shaders"), which allowed the CPU to send simple instructions, and then the GPU would calculate or modify the model on the fly, allowing it to make Models more or less detailed depending on how far they were from the camera. Technically Tesselation Shaders are different, but I honestly don't understand the difference.

2008-2018 - Unified Shaders

By 2008, GPUs had transformed into just thousands of tiny processors each capable of running a sequence of shader programs, fully customizable. Each processor is ~10x slower than the CPU, but even low-end integrated laptop GPUs have ~96 cores. High end integrated graphics have ~384 cores, and the GeForce RTX 4090 has 16,384 cores. At this time, people began using these to calculate things besides graphics, including, bitcoin mining, and also Artificial Intelligence.

2018-Present - Realtime Ray Tracing + AI

And now we finally get back to something involving 3d lighting. GPUs added specialized processors for calculating Ray Tracing, which works almost 100% differently.

• The GPU imagines a screen with a low resolution, often about a quarter of the real resolution.
• The GPU imagine tens or hundreds of "rays" shooting out of each Pixel. The closer to perfectly straight they are, the more "important" they are.
• The GPU calculates which triangle each would hit, and where on that triangle.
• The GPU calculates the texture color for that point, and the "shininess". It also calculates the normal vector.
• The GPU "reflects" the ray over the normal vector, tints it a color pased on the texture color and shininess, separates into tens or hundreds of rays, and repeats, with an adjusted "importance" depending on the shininess. Some implementations may deliberately try to aim some of the reflected rays at light sources, which can greatly improve performance and quality, but greatly interferes with proper reflections, which kind of defeats the point of Ray Tracing, unless you do some tricky statistical math to compensate.
• This continues until a ray hits a light source, or it runs out of time.
• The GPU takes the weighted average color for each ray depending on their importance.
• This results in a very dark and fuzzy image, so the final step is to use AI to brighten the image, sharpen the edges, and upscale it to the final resolution.

Ray Tracing can produce perfect curves, since it's not actually limited to triangles, and also produce incredibly realistic reflections and lighting, which are the focus of your question. But they're incredibly processor intensive.

Another note is "volume tracing", where rays are allowed to go through semi-transparent triangles, and may emit additional rays as they move "through" things, giving realistic smoke, clouds, or glass.

Obviously, most rays won't actually hit a light source, so many implementations add some variant of "Photon mapping", where they first emit rays from the various light sources into the scene, which reflect around like normal ray tracing, and keep track of how much of which colors of lights are in each general area, in order to "light" a scene. After doing this some small number of bounces, they then do normal ray tracing. This is more complex to design, and takes a lot more memory to store the lighting information, but results in rays detecting light dramatically more often, and greatly reducing noise, and thus quality. Due to the complexity, a lot of implementations instead just calculate a "shadow map" with only a tiny number of bounces, once every few frames, which works fine as long as lights don't move fast while close to the camera.