How does 3D software like Blender, Maya, Unity, or Houidini etc. render the effects of lights illuminating a 3D scene?
Not all of the software you listed calculates lighting the same way, or the same in all circumstances. There are a few different approaches:
Offline raytracing / path tracing are used by 3D modelling software like Maya/Blender/Houdini when you want to output a high-quality final render, or a frame for something like an animated film.
In these techniques, the software traces the actual paths taken by each ray of light departing a light source or arriving at the camera, as it bounces around your scene or gets diffused by your materials.
This can produce highly accurate, photorealistic results, including detailed reflections, but it can be extremely performance intensive. A single frame might take seconds, minutes, or even hours to compute in full, depending on the complexity of the scene and the target resolution/quality. 3D film studios and effects houses frequently have render farms full of high-powered computers to chew through all the raw computation required.
So this often can't keep up for the purposes of realtime interaction, and the apps will fall back on iterative approximations or rasterization techniques to keep the editing views and preview panes responsive.
Realtime rasterization is the family of techniques at the heart of most modern games and interactive 3D apps.
It flips the question from "for each ray of light, what surfaces does it hit in my scene?" to "for each surface in my scene, how does the light illuminate it?"
Typically we draw the scene triangle by triangle of each mesh, projecting the corners of that triangle onto the screen, then filling each pixel between those corners with a colour.
To find the right colour, we run what's called a pixel shader or surface shader, a small program that your GPU executes for every pixel it draws. This program figures out which way that bit of the surface is facing — based on the mesh's normals, or a normal map texture it looks up into, then evaluates how much that direction is pointing toward or away from the light source. Places where the surface points directly at the light get drawn bright, while places that face perpendicular or away from the light get drawn black. (Plus some extra considerations for specular reflection that depends on the view direction)
By adding up these colours, computed for each light source in range, we get the net illumination of the pixel.
The trouble with this approach is it's only an approximation of the most direct light path from the light to the surface to the eye. It doesn't take into account anything else in the scene — like objects that sit between our triangle and the light source and should block the light, casting a shadow, or nearby illuminated surfaces that should bounce light onto the surface we're drawing.
So we have to approximate these effects, like drawing a shadow map to estimate which scene pixels are actually occluded from the light's perspective, or adding ambient terms or other tricks to simulate bounced light and reflections. Modern rendering in games uses many sophisticated approaches to fill in for these interactions and get a more natural look.
This rasterization technique can get us fast, plausible results for a high framerate, but it can sometimes show artifacts that can reveal the trick or make the result look lower quality than raytracing.
Hybrid "baked" solutions are often used to increase the fidelity of lighting in realtime apps and games, while keeping framerates high.
In these models, an offline "baking" pass uses raycasting-style techniques to calculate bounce light interactions in the scene, then stores that information to a light map or lighting/reflection probe data structures. This can also be done at runtime, by an asynchronous background thread that updates slowly-changing illumination information periodically.
At runtime, light that can change quickly (as on moving objects or lights that move and change) is rendered with traditional rasterization techniques so it updates instantly, while the bounced light or light on static objects is looked up from the data structures. This sampling process is often lossy and not perfectly accurate, but it can go a long way to make the scene look more like an offline render, with convincing bounced lighting.
Realtime ray-tracing is an emerging technique leveraging new and more powerful graphics cards with built-in hardware acceleration for common raycasting approaches. While it's still out of reach to render a AAA game completely with raytracing at an acceptable frametate, games can use this hardware selectively, to do things like mapping more convincing reflections on shiny surfaces.
Raymarching is a more exotic technique not often seen in mainstream games and apps, but popular in "demoscene" creations or playgrounds like ShaderToy. In this technique the content of the scene is expressed as a signed distance field, and for each pixel we draw we "march" a ray through this field, one step at a time, until it hits something.
It's conceptually similar to raytracing, but expressing it as an iterative steppinh through a distance field lets us leverage conventional rasterization hardware to accelerate it on the GPU.
It's difficult to use with conventional animated meshes, but can be effective for games that use procedural geometry, like Claybook.