Pixel Picking vs Raycasting

Question

I was recently introduced to a way of picking in OpenGL known as pixel picking (or color picking).

From my understanding, how this is done is that the scene is rendered to the back buffer with a different rgb color representing each object. By reading the color of the pixel clicked by the cursor, the relevant object can be picked. The immediate issue I can think of when using this method is that during the picking process, the scene have to be rendered 2 times, once for picking and another time for display. Additonally this also limits the it to 256^3 unique objects for picking.

With this, I would assume this is a much more accurate way of "raycast" than to use physics raycast. By placing a camera at the raycast source, render once for "raycast" picking, and resetting the camera to the original position for normal render. This would reduce issues of "bullet hitting invisible walls" as its not dependent on physics collision, which can be affected by convex hulls for complex meshes.

Is there any reason why this method is not used? Could it be over complication? Or perhaps it is what is happening behind the scenes in raycasting?

---

As an example where pixel picking might be better would be when raycasting through a wire fence. Most wire fence models are just textures with a transparent channel, giving the illusion of a wire fence while saving on geometry by having 1 single plane. Using color picking, what the player see is what the camera picks. Whereas in raycast, the raycast will not shoot through the plane unless it doesnt collide with the plane at all.

Answer 1

Pixel picking is effective if you need to do lots of raycasts from one source. You bear the overhead of rendering and readback once, then get all your many picks at low incremental cost each. To get just one ray result you've already had to calculate thousands or millions whether you use them or not.

If you're only doing a few raycasts at a time (say a handful of bullets or IK placements per frame), or you need to do casts from multiple sources/multiple disparate directions (say, multiple characters who each need to shoot from their viewpoint) then pixel picking gives you relatively little bang for your rendering buck.

A lot of games fall into this latter category, where they need small groups of raycasts all over the place rather than hundreds of raycasts from one source, so doing the raycasts individually with physics is more efficient for them.

Doing the physics raycast also lets you get access to the impact normal and high-resolution UV/position cheaply, which may cost extra storage & bandwidth to extract in a pixel-picking strategy.

Lars Viklund raises another important limitation: quite often gameplay raycasts like bullet shots are performed on the server to maintain fairness in multiplayer games. Servers often run without rendering so they focus strictly on processing gameplay logic as fast as possible. Making gameplay dependent on rendering may complicate or bog down the server code in these contexts.

For the chain link fence example, most games solve this though physics layers — categorizing permeable barriers like this as "see-through/shoot-through/no-nav-through" so raycasts for detection and shooting go through it but navigation/movement/collision checks still treat it as solid. If you need pixel perfect results around this kind of obstacle from the player's point of view, you might be able to check the raycast result versus the depth buffer, without a separate rendering pass.