Of course, I see the benefit in using polygons to create 3D worlds. But I sometimes wonder if other methods exist and have been used in published computer games; perhaps even hybrid methods of 3D animation that combine polygons with some other style of rendering, so that the limitations of polygonal rendering might be overcome or just mitigated. For example, a way of rendering human bodies without jagged limbs.
A technique which comes around again and again for decades is raytracing. While usual 3d engines are based on taking a bunch of vertices in 3d space and see where they end up if transformed to 2d space through a projection matrix, raytracing engines simulate a ray of light from each screen-pixel into the scene and see where it hits a surface.
Raytracing engines usually like to show off by rendering perfect spheres reflecting their environment. While spheres and reflections are hard to do well in polygon-based engines, they are quite trivial to do with raytracing.
Unfortunately GPUs aren't optimized for raytracing, so very few games make use of this extensively, so no GPU vendors bothers to optimize for it. This chicken-egg problem prevents the method from becoming mainstream. There are a few experimental games which use raytracing-based engines, though.
To throw out one other possibility, we can define a scene in terms of a signed distance field and raymarch against that. Many demoscene creations use this technique to create highly detailed scenes on a minuscule data budget, letting the mathematical formulae do the heavy lifting.
This scene, from a demo called rgba slisesix was created by Íñigo Quilez, one of the most prolific creators using this technique. It's generated from a 4 kilobyte executable, with no models or texturemaps.
(Check out Íñigo Quilez's site for more examples and fascinating articles about the math & programming behind these and other effects)
The technique is darn near magical in its ability to represent smoothly curving surfaces and absurdly, even fractally fine detail as you zoom in. Thanks to some clever space-folding math, it's also great for representing repeating content, like regular structures in architechture. Once you render one motif/column/arch/etc, you can mirror and tile it infinitely at little additional cost.
There are many downsides though:
The distance functions for a complex scene can be very expensive to evaluate. (See the elephants above. As a polygonal mesh scene, we could render a similar visual at a much higher framerate using current hardware & software) The usual performance fixes we might apply to polygonal scenes, like level of detail reduction, culling, and impostors, don't have a direct or easy-to-plug-in analogue in signed distance fields.
Authoring a signed distance field scene requires a kind of mathematical sculpting skill that few game content creators have mastered. It's not as simple as creating your scene in a conventional 3D package and clicking "convert to signed distance field." While voxelized distance information from a mesh can be used for some effects like shadowing, any demo with complex signed distance visuals has almost certainly been built from the math up.
Modifying a scene dynamically is even trickier. With a polygonal scene, we can potentially move, delete, or add each polygon independently. We can used skinned meshes to animate characters. With a signed distance field, everything is a product of the formula, and not every intuitive operation we might want to apply has a clear or side-effect-free expression in that domain.
So, thus far polygonal rasterization has proven to be a more flexible tool, practical for the majority of the games we build today. But I bet there could be some clever ways to design a game around the particular quirks of distance field raymarching and create something pretty cool!
One interesting development in that direction: as I understand it, MediaMolecule's Dreams uses signed distance fields for their creation tools, but they don't render them directly - they convert them to points or quads to render the resulting forms in a more conventional fashion.
It's possible to build a graphics engine using other geometric primitives. In 1994, Accolade released a 3D fighter called Ballz used spheres. It was impressive for the hardware at the time & notably different from other things available on console, but didn't do particularly well as a franchise. The underlying engine was later used for a number of virtual pet games.
A more recent example is Toribash, which appears to use a combination of spheres, cylinders, cones & polygons.