How does Unreal Engine 5's Nanite work?

Question

I have seen Unreal Engine 5 showcasing their Nanite system a long time ago. Like everyone, I was impressed. Since this is something that would revolutionize the gaming industry, I expected it to come into the hand of the public, by them giving it to us or leaking it out. But so far, I haven’t been able to find it. So, I jumped on Stack Overflow to ask this question, hoping to find someone who knows how it works and are willing to share it with the community.

Answer 1

I'm no expert, but since no one else has jumped in, I'll do my best to summarize what I gathered.

First, here are the descriptions of Nanite directly from the Unreal folks:

• "A Deep Dive into Nanite Virtualized Geometry" at Siggraph 2021 by Brian Karis with slides here. This is the primary resource I used for my answer below.
• "Inside Unreal" has a more casual conversation with Brian about Nanite.
• "The Journey to Nanite" also by Brian explores the alternative approaches they considered.
• "Bringing Nanite to Fortnite Battle Royale in Chapter 4" describes their effort to port Fortnite to use Nanite.
  • Very related are their blogs on bringing virtual shadow maps and Lumen to Fortnite.

How Nanite Works

GPU-driven rendering

First off, it's a GPU-driven rendering pipeline meaning culling (frustum and occlusion) and level-of-detail (LOD) selection happens on the GPU using compute shaders. Compute fills vertex and index buffers which are then rendered using indirect draw calls. For more see this talk from the Ubisoft folks ~2015 following Assassin's Creed Unity. Traditionally, the CPU would cull and select LODs for the frame then use draw calls to draw each visible mesh. Note that Unreal Engine 4/5's default renderer is also GPU-driven rendering.

128-triangle clusters hierarchically arranged by LOD

Next, in traditional rendering systems you have different LODs per-mesh, and your artists are heavily involved in creating good per-mesh LODs.

In Nanite, LODs are instead determined at a sub-mesh level with much less artist involvement. A whole tree (well a directed acyclic graph, or DAG, technically) of 128-triangle clusters (also called "meshlets") is computed for each mesh at asset import time. Each node in the tree is <=128 triangles and the children of any node represents a more detailed view of that node. Then a "cut" of this tree is determined at runtime (on the GPU!), and based on this "cut" a set of triangle clusters will be rendered. Culling also happens on a per-cluster not per-mesh basis. Here's an excerpt from those slides:

A stable number of triangles per frame with little overdraw

When all this comes together well, you end up with a system with a very consistent number of triangles rendered to the screen per-frame (~25M for their demo scene) and very little overdraw (aka wasted shader work). This efficiency is what lets Nanite achieve crazy high detail at decent FPS.

Note that Nanite does not currently use any raytracing, although they mention this might change in the future.

The devil is in the details

Now this glosses over a lot of really hard details! If you implement such a system naively, you end up with really bad cracks or seams between clusters especially at adjacent jumps in LOD level >1, visual popping as you switch between LODs, memory issues as this is an enormous amount of data (each mesh's full tree of clusters), and performance issues since this involves a ton of GPU compute pre-shading work! And exactly how does this cluster-based visibility culling happen on a GPU?

Mesh simplification and constructing the heirarchy

First, a ton of work went into determining how triangles should be clustered and hierarchically arranged at asset-import-time so that you don't see cracks and yet you can still efficiently determine a cut of this hierarchy at runtime. This involves complex graph partitioning and multi-dimensional optimization, see the slide 50 for more.

It's worth pointing out that the most detailed view of a mesh in Nanite (the leaf nodes in the cluster graph) are exactly the same triangles as the original asset. Nanite doesn't optimize away any details, instead it displays simplified triangle clusters (higher nodes) only when the detail change isn't visually perceptible.

LOD N vs LOD N+1 difference/error calculations

Also very essential and related (and if I understand correctly this is one of the most novel contributions of Nanite) is calculating good perceptual error metrics between different LODs. Basically, you need to know how much worse a simpler triangle cluster is than its child (i.e. more detailed) clusters in order to do good LOD selection. Amazingly, if the error difference is <1 pixel, and you have some temporal anti-aliasing on top, you won't notice any popping!

Cutting the cluster heirarchy DAG (LOD selection)

These two, a good cluster hierarchy and LOD error calculations, come together at runtime with view-based information (in GPU compute, using a bounding volume hierarchy (BVH) and custom parallel task system) to determine the "cut" of the cluster DAG, which is the LOD selection for that frame.

Cluster-based visibility culling

For visibility culling, they use a modified "two-pass occlusion culling" technique where you use what was visible in the previous frame (captured in a hierarchical z-buffer (HZB)) to massively speed up your determination of what is visible this frame. Nanite diverges somewhat from other two-pass culling systems because they use a bunch of information from the LOD selection phase described above to make this work. The output of this visibility check is then written to a "visibility buffer" that includes per-pixel depth and cluster index data.

Streaming virtual geometry

For memory management, they aggressively eject unused clusters from working memory and stream in new ones from disk. For this reason, it seems a decent SSD is basically required for Nanite to work. They call this "virtual geometry", analogous to "virtual texturing". Formatting, compressing, and deciding what to stream is complex. About ~1M input asset triangles becomes ~11MB compressed Nanite data on disk (slide 144).

Material selection and shading

Once they have this visibility data, they then need to do material shading which outputs to g-buffers and the rest of their deferred shading pipeline. One of the more illuminating slides for me was this description of their per pixel material shading:

Which did seem crazy to me, but they point out that they get a very good cache hit-rate and no overdraw.

Knowing which materials are visible and which pixels they are assigned to is another complex task, and they use a combination of repurposed HW depth-testing for material testing and screen tiles for material culling to do this. See slides 98 onward.

Rasterization woes

It's also worth mentioning that because Nanite has such a crazy high level of detail, they ran into rasterization problems. (Rasterization is the process of matching triangles to pixels). They commonly had triangles as small as pixels, which preformed poorly on the built-in hardware rasterization, which is overwhelmingly the common way to do rasterization. So they wrote their own software rasterizer (called Micropoly?) and it runs in a compute shader. This includes doing their own depth-testing to create their z-buffer. They then chose between HW and SW rasterization per-cluster (big triangles still work better on HW.)

Relatedly, because triangles are so small, UV derivatives need special treatment (slide 106-107).

Virtualized shadow maps

Next, there are shadows and multi-view rendering. I feel less confident summarizing this, so I'll refer you to slides 115-120, and also their blog about bringing virtual shadow maps to Fortnite6. Apparently their unique triangle cluster approach lets them efficiently maintain high resolution (16k) "virtual shadow maps".

Limitations

Folliage

My understanding is that Nanite isn't great at traditional alpha-clipped foliage like leaves and grass, although that seems to have improved in 5.1 16 17. In the slides, they mention limitations of the software rasterizer as a key problem here.

Their blog about bringing Nanite to Fortnite5 describes a lot of the challenges with vegetation and ultimately they decided that their grass and leaves would be formed out of geometry instead of alpha-clipping.

Tiny instances

And they call out tiny mesh instances (~1px in size) as a problem they're still working to solve, with hierarchical instancing being their chosen direction. Currently they use an imposter system. As an example, if you have a large building made of tilable wall segments and you zoom out such that these wall tiles are ~1px, Nanite doesn't perform well currently.

In 5.1 with their work to bring Nanite to Fortnite5, they introduced a "Preserve Area" flag which helps with this a bit.

??

They also mention on slide 94 that "the reliance on previous frame depth for occlusion culling is one of Nanite’s biggest deficiencies", although it's unclear to me all of what that implies.

Nanite goals, alternatives, and prior work

Goals

At the start of their slides, they discuss Nanite's goals and other approaches they dismissed. The goal was to be able to render high fidelity assets without require a ton of up-front work by asset creators and instead let the engine dynamically change the level of detail to maintain real-time performance.

Alternative approaches

Approaches they considered and dismissed for various reasons included: voxels (bad at hard surfaces; fundamentally uniform sampling), subdivision Surfaces, displacement maps, geometry images, and point rendering.

This keynote by Brian at High-Performance Graphics goes into a lot more detail about these alternatives and why they didn't fit Nanite's goals.

Prior work

Nanite was built on a lot of prior work, some of the most important seems to be Quick-VDR (2004) and Batched Multi Triangulations (2005), but there's 85 other citations on slide 149 onward.

Who made Nanite?

The main Epic folks behind Nanite seem to be (based on the slides) Brian Karis21 22, Rune Stubbe23, Graham Wihlidal24 25.