Why are 3d projections on a 2d screen not like what the human eye sees?

Firstly, I'm sorry if this question doesn't make much sense or is poorly written - I have almost no experience with programming, and none at all with game development. I'm also not sure it's an appropriate question for gamedev stack exchange, so sorry if that's the case.

Anyway, the question is this - what are some ways of doing the vision in a fps game?

If I understand it correctly, the problem can be stated the following way - given a 3d environment given in 3d coordinates, and a camera with some direction and certain field of vision, our goal is to somehow project the environment, to a 2d screen, similar to how our eyes do it, or similar to how a camera does it.

One way to do this is the following - whatever is exactly in the direction of the camera will get displayed exactly to the middle of our 2d screen (let's presume the 2d screen is round for simplicity, with radius 1), and the rest in the field of vision will get projected linearly based on the angle. For example, if field of vision is 90 degrees (anything 45 degrees from the direction of the camera will be in the field of vision), then things 45 degrees away from the direction of the camera will get displayed to the outer circle/edge of the 2d screen, and things 22.5 degrees away will get displayed to the circle with radius 0.5. This might be similar to how a camera or our eye does it, but I'm not sure.

Now in reality, this doesn't seem to be how it's done. To give an example, if I take something like DOOM, or I think any modern fps game, and I were to stand in front of a very long wall that's of the same height everywhere, and I were looking perpendicularly towards it, the height of the wall will be the same everywhere in my 2d projection - in reality though, or a camera for example, would display the very furthest (furthest from the center of the vision) parts of the wall to be shorter in height. But, given a small enough field of vision, the difference seems negligible.

Either way, it seems like fps games don't really consider this fact, and I want to ask - why? Maybe the difference is barely noticeable, but much more power demanding? Either way, I would appreciate an explanation or information concerning how exactly the human eye or camera projects 3d environment to a 2d picture.

If we're willing to approximate the viewer as a cyclops whose single eye has a pinhole pupil (and this turns out to be a much better approximation than it sounds like - more on that later), then I'd argue that both projection methods you describe are correct. For different shapes of screen.

Thinking of the problem angularly gives the correct result if the screen is a spherical dish, hemisphere, or globe centered on the viewer's eye.

As the viewer sweeps their gaze around the image (taking care to keep their pupil at the exact center of the sphere to not upset the theoretical mathematician who put them here), each degree of travel translates to a constant length across the surface of our curved screen.

Since the entire monitor's surface is the same distance from their eye, no part of the image is foreshortened or otherwise scaled/distorted from their perspective. So as the infinite wall recedes away from them, taking up an ever narrower arc of their visual field, it must also span an ever smaller height on the screen, until eventually its top and bottom lines - which had been parallel dead ahead - meet at a sharp vanishing point 90 degrees to the left. The lines have to be curved to make this work (specifically, straight lines in the rendered world will map to great circles on our screen. Groups of parallel lines will map to great circles meeting at a common pair of diametrically opposite poles, like lines of longitude)

Now, I don't know about you, but I don't own a spherical screen. When I play games, they're typically on a flat screen. Which means it's not a constant distance from my eye at every angle: as a feature recedes away to my left, so does the screen itself!

I'll spare you the equations for now, but if our monitor/viewport is exactly parallel to the wall, and we use linear perspective to draw the wall at exactly the same height on the screen all the way across, then from the viewer's perspective it will shrink in visual angle in exact proportion to the virtual wall it represents, as the two recede into the distance to the left.

The two parallel lines never need to meet at a point in this view, because even the absurdly-widescreen monitor I've drawn in the diagram above still has finite width. We'd fall off the edge of the screen long before we ever reached the vanishing point (infinitely long before, in fact!) If we did somehow have an infinitely wide flat screen, we'd see the vanishing point arise naturally - not because it's built into the projection of the world onto our screen, but because the projection of distant screen in our eye affects lines on the screen in exactly the same way any other lines in the world, including the original wall we're rendering.

Of course, if we turn our viewport (game camera) so that it's no longer parallel to the wall, then linear perspective will replace those parallel lines of the wall with ones that literally converge to a vanishing point on our screen, as we'd expect, and again the net effect of the size of the feature on screen and the distance to the screen itself will exactly match the right amount of shrinkage/foreshortening for a given angle.

This is because both projections work by considering where a line from the viewer's eye to each feature point on the object (the cyan lines in my diagrams above) intersects the surface of the screen, and that's where they place the image's corresponding feature point. Since these lines point directly out from the eye, they don't travel across our visual field - each one maps to precisely one point in the 2D space of our vision, no matter the depth. So as long as we draw each feature on the same "eye ray" as its original, it will be placed in the correct part of our visual field for whatever kind of screen we're intersecting with / displaying on.

To see the equivalence another way, imagine nesting these two diagrams - so the spherical screen is displaying an image of the flat screen displaying an image of the wall, or vice versa. I used the same eye rays to feature points in both cases, so they'll line up exactly, and we get the same image on the spherical screen using angular rendering whether we're trying to draw the wall itself, or the image of the wall we captured on the flat screen (if we could extend it to an infinite flat plane).

So, neither linear perspective nor this spherical/angular projection is a "more correct" way to project the scene, they just depend on our rendering intent. The spherical projection has to work a little harder to "bake in" all of the perspective shrinkage effects for every screen normal direction instead of just one normal in the linear case, though in exchange it's able to render a complete wraparound view if you're lucky enough to have hardware that can display it. :)

So, using linear perspective on a flat monitor is no more approximate than using spherical perspective on a spherical monitor. They're both geometrically correct for the simplified viewer we set out to model.

It's not necessary to model the roundness of the retina, the angular swivel of our gaze, or the distortion of our lenses in the image we render on the screen, because you're already viewing the screen with your angularly swiveling, lens distorting, round-retina real eyes! So all that gets applied as a "post-post effect" by our own physiology. ;)

All we need to do - in either model - is correctly plot on the screen what you'd see if your view ray continued "through" the surface of the monitor in a straight line and hit the corresponding point in the virtual world on the other side.

Now of course, having two eyes and the ability to move in space complicates this - a single image will be correct from only one viewpoint. So in non-stereoscopic games we typically choose a compromise viewpoint that's "good enough" - and the approximation of the projection comes from that choice of viewpoint, not from the perspective projection math that uses this that viewpoint as an input. Stereoscopic games for VR headsets or the 3DS have better ability to locate the player's actual eyes relative to their display and show each one a custom-tailored image, so they can get much closer.

The last detail is that our pupils aren't infinitely narrow pinholes, so we can only focus on a single focal plane at a time, and experience depth of field blurring closer and further away, in addition to lens aberration effects. This affects the sharpness of features in our visual field, but not the location of the center of their circle of confusion, so again it's not a perspective projection approximation, and the pinhole cyclops turns out to be a pretty decent approximation for figuring out where to draw stuff. XD

One last note:

There had briefly been a comment below asking about the stretching we see at the edges of the screen in some games using linear perspective. Although it's deleted now I think it raises a point worth describing in a bit more depth:

As mentioned above, both of these styles of projection are only geometrically correct when viewed from one point, where all those "eye rays" intersect.

Since most games don't do head-tracking to move this point dynamically to match your actual position (though the effect can be dramatic and extremely convincing when they do!) they tend to pick this viewpoint based more on the aesthetics of the game and the informational needs of the gameplay.

Shooters tend to push this further, since having more peripheral vision available can be a life-or-death matter in these games. So they'll choose a large field of view value for their camera - that corresponds to a hypothetical viewer whose eyeball is very close to the screen, so they have to rotate their eye further to sweep their gaze from center to edge. When viewed from this idealized close position, the extra stretching at the edges of the screen gets foreshortened by the extreme angle at which we're viewing it, and the effect is correct - like standing in the right place for an anamorphic chalk painting, the perspective of the screen and the image on it exactly compliment each other and the scene snaps into correct perspective.

But if we move back from the screen to a more comfortable play position, we're bringing more of its area into the narrow forward cone of our vision, where it's more perpendicular to our view and less foreshortened than the math was made for, and the distortion becomes more apparent.

Fortunately, our brains are pretty malleable, and players will tend to get used to even fairly significantly exaggerated FoV values when playing on a flat screen, so we tend to mainly notice the effect in stills. This does not go the same for VR though, where matching the field of view to the actual device viewing conditions is critical for maintaining player comfort, so don't push your FoVs everywhere. ;)

• That's due to the fact that games typically use a wider FoV angle than is geometrically appropriate for the distance of the viewer from the screen. It's a correct perspective from a viewpoint much closer to the screen, where those stretched regions would be foreshortened further by the more extreme viewing angle. By viewing it away from that geometrically correct vantage point, we perceive distortion, like looking at an anamorphic drawing off-angle. A similar effect happens on the spherical screen too if we view it from outside the center, like the perspective of our diagram. Commented Aug 11, 2017 at 6:33
• "If we view it from outside the center" - does that happen with the eye? NO. Since a raytracer has a "pinhole" centre and all the rays are cast from that point, is it more like the eye? YES (only, in reverse since the eye receive photons instead of casting rays). Commented Aug 11, 2017 at 6:37
• For most games we don't have the option of rendering directly into the player's eye. We render onto a screen, and the player can move their head and eyes relative to that screen. No matter what projection or screen we choose, if the player looks at it from a different vantage point than the one we used to generate the image, they'll perceive distortion. Commented Aug 11, 2017 at 6:41
• This should be the accepted answer instead... I knew the difference between the two methods before, but never realized that the flat screen one is better than spherical screen one for, well, flat screens. Commented Aug 11, 2017 at 17:03
• +1 Excellent answer. Thanks for taking the time to write it. Commented Aug 11, 2017 at 21:58

Your assumption is incorrect, in that things farther away from the viewpoint are, in fact smaller. This is called "perspective."

Given an infinitely long, perfectly flat wall normal to the view vector - that is, you're looking straight at it, perpendicular to the wall surface, and given that you could see to infinity in each direction, the wall would appear to curve, and the top and bottom would meet at a point on the left and right.

This is related to the non-linearity described by trig functions, specifically tangent. As the angle between the view vector and rendered pixel increases, the relative (world space) distance between adjacent rendered pixels increases, or, more accurately, the volume of the frustum described by the pixel bounds increases.

(TL;DR: adjacent pixels at the edges of the display contain parts of the world that are farther apart - or if you prefer, cover more area of the rendered volume - than adjacent pixels at the center of the view.)

Google "two point perspective" and sketch some stuff out for fun.

The big reason that we can't make stuff on the screen look like it would in real life is that, in most cases, we don't know how far your face is from the monitor, or in what relative position. Given that information, the screen would appear to be a window - the glass kind, like in your house / apartment / whatever - rather than a flat display.

This is one of the (few) great things about VR. Given the position / orientation data, we can render things the way they should be rendered. Stereoscopic is a gift-with-purchase.

UPDATE

In retrospect, I think @DMGregory's answer is more accurate, and certainly more informative. I'd delete my post, but I think the discussion-in-comments (I'm sure that'll get flagged for deletion in a few years) is useful, so I'll leave it for the time being.

• I thought that's what I said - "the wall will appear to curve" is how it should be. But I don't think I've ever played an fps where the wall wouldn't appear to perfectly parallel, across the whole field of vision. Commented Aug 10, 2017 at 23:26
• @JohnP That's because your field of view (FoV) is quite small when looking at a monitor, and we're doing the best we can based on very limited data. (Really, most of us copied-and-pasted the code for calculating a projection matrix 20 years ago and haven't thought about it since. Stupid W.) However, with a sufficiently large level, you'd see the curve given the standard camera setup in Unity, Unreal, MonoGame or anything else. It's a side-effect of the way the projection matrix is calculated, which is based on, well, reality. Do the walls in your house look curved to you? Commented Aug 10, 2017 at 23:30
• No, Unity and other modern engines do use linear perspective, which will not bend straight lines, and this is actually mathematically correct when rendering an image to display on a flat screen. Ignoring lens distortion for the moment, straight lines only curve in the image when the image plane itself is curved - like rotating your phone camera while capturing a panorama. Take any single photo from the sequence and the lines will be straight. It's the attempt to cram all those planar views into a single wraparound image that introduces curvature. Commented Aug 11, 2017 at 3:21
• Put the visual cortex aside for a moment, and just think about how light hits a camera's flat CCD sensor. If we neglect unwanted distortion due to the glass of the lens itself, a straight line in the world will map to a straight line of pixels on the sensor. This is not just due to the small size and angles involved - it holds true even as we extend the image plane wider, like a room-sized camera obscura. We'll see curvature in our visual field when we look at such an image, because the plane recedes away from us to the sides, so we needn't bake curves into the image itself to look right Commented Aug 11, 2017 at 13:08
• Consider this screenshot for example: link. The horizontal FoV of this picture is 170°, but the wall edges are pixel-perfectly straight. Commented Aug 11, 2017 at 17:16

Others may come with a more complete answer but the gist is:

3d computer generated images are displayed on a flat plane (your monitor), so the brains who developed the science of graphics programming came up with this technique called 'perspective projection'. This technique basically takes all the vertices that are in the viewing frustum:

...and 'squishes' them to make them flat on a single plane, which is the viewport. The vertices composing the objects are projected on the viewport in a "direct" fashion, not curved whatsoever. This is why everything looks straight.

This technique is both good enough and fast enough.

But that's not how the eyes work. What we see is not projected directly to a flat plane but directly on our retinas.

• "But that's not how the eyes work. What we see is not projected directly to a flat plane but directly on our retinas." Do you somehow view your computer screen without projecting its image onto your retinas? ;) Baking the effects of eyeball physiology into what we put onto the screen would be double-dipping, applying the effects twice - once on the screen, and a second time in our actual eyes - which is most likely not the look we're aiming for. ;) (It makes sense for DoF effects though, since conventional monitors are all in one focal plane, so out-of-focus blurring needs to be baked-in) Commented Aug 11, 2017 at 3:12

You may be interested to know that raytracing, by its nature of being more similar to the eye than your typical flat-projected computer renderer, does in fact account for the fisheye effect, which is the term you're looking for. You may not be a fish, but all creatures with (round) eyes experience it to a greater or lesser degree. For example:

Interestingly, this is often considered to be an undesirable effect in raytraced rendering. The reason is that when a human being views such a space, the visual cortex actually normalises the resultant image such that your vision does not at first glance (snort!) appear to have this effect... unless you examine it carefully and explicitly. This is one of several such "post-processing" effects on our vision, some of which I believe lead to a portion of the famous optical illusions we all know so well.

As others here have explained, the details of why a common present-day renderer does not account for the fisheye effect is based on their method of projection, which is entirely different from casting thousands or millions of rays outward in a convex semi-hemispherical arc - which leads to fisheye.

• Distortion like this is not revealing a more "physically accurate" way to depict the scene, it's rendering for a curved display surface instead of a flat one (often unintentionally), by using distances along the ray instead of perpendicular to the image plane. You can verify this by holding your head still in front of a window, closing one eye, and tracing the outline of a building across the street onto the glass with a whiteboard marker. Straight lines in the world will map to straight lines on the glass, just like Brunelleschi's experiment. Commented Aug 11, 2017 at 3:00
• @DMGregory It absolutely is more physically accurate (perhaps I should qualify that term - more lifelike in terms of its similarity to the operations of a physical eye) to take samples around the orb than it is to use the sort of flat projection that GPU-based renderers typically do, which is cheaper but also far less physically similar to eyes in their means of representation. Commented Aug 11, 2017 at 5:38
• @DMGregory You misunderstand Brunelleschi's results. There are reasons such an experiment produces apparently straight lines and that is because of the scale of the object, the relative narrowness in your field of vision when head is kept still, and nearness of the window pane. See e.g. here. They state the human eye does a great deal of correction. Commented Aug 11, 2017 at 5:50
• If we render to a curved screen, it's accurate. But screens are typically flat. Addressing pixels like they were arranged about an orb doesn't match how the image is being viewed. The lack of distortion we see in Brunelleschi's case is not due to narrow FoV, but because the display surface is flat. This is echoed by your link: "When a wide-angle photo is displayed, the pixels are not shown at the viewing angle of original capture, but are instead compressed onto a flat rectangle in front of the viewer. If the image was projected onto a hemispherical screen…it would not look…distorted." Commented Aug 11, 2017 at 5:52
• We're not taking the human eye out of the equation. Whether I'm looking at a render of a building or a real building, both those light fields are going through my round eye. So we don't need to bake the roundness of the eye into our image any moreso than we need to bake it into the building. A straight line of a wall and a straight line on a flat drawing of a wall both get processed correctly by our visual system. Since we're not rendering directly onto the retina, but onto a monitor, it's the flatness of the destination screen that shapes our projection, in both raytracing and rasterization. Commented Aug 11, 2017 at 6:15