Toon/cel shading with variable line width?

Question

I see a few broad approaches out there to doing cel shading:

1. Duplication & enlargement of model with flipped normals (not an option for me)
2. Sobel filter / fragment shader approaches to edge detection
3. Stencil buffer approaches to edge detection
4. Geometry (or vertex) shader approaches that calculate face and edge normals

Am I correct in assuming the geometry-centric approach gives the greatest amount of control over lighting and line thickness, as well eg. for terrain where you might see the silhouette line of a hill merging gradually into a plain?

What if I didn't need pixel lighting on my terrain surfaces? (And I probably won't as I plan to use cell-based vertex- or texturemap-based lighting/shadowing.) Would I then be better off sticking with the geometry-type approach, or go for a screen space / fragment approach instead to keep things simpler? If so, how would I get the "inking" of hills within the mesh silhouette, rather than only the outline of the entire mesh (with no "ink" details inside that outline? (AKA suggestive contours, creases).

Lastly, is it possible to cheaply emulate the flipped-normals approach, using a geometry shader? My concern with this is that I could certainly duplicate every single vertex and scale these accordingly, but how would I approach flipping normals, and distinct colouring in the fragment shader?

What I want - varying line thickness with intrusive lines inside the silhouette...

What I don't want...

---

EDIT: Further research has turned up the following...

Since I have a massive vertex count on terrain, even considering distance-based LoD, neither flipped-normals nor a geometry shader based approach (even with frustum culling) would be a sensible option due to sheer computational complexity involved in duplication & scaling of all uploaded vertices.

Considering that I don't need per-pixel lighting in the form of solid-tone shading on the terrain surfaces, it also becomes less prudent to consider any face-normal based approaches -- otherwise a requirement for correct surface lighting -- as these are naturally quite expensive to calculate. It is however true that they give the finest degree of control; for example, the ability to shade edges using "artistic" strokes: Beautiful, but again, not really viable for a massively complex game environment.

Stencil buffers are something I'd rather avoid as I'd prefer to do all work in shaders. (The above example with the red outline was done with a stencil buffer -- old school.)

This leaves fragment shader image-space approaches. Computational complexity is reduced to the number of fragments rather than the number of vertices (in my case, this is 10-100x fewer operations than I would have to do in the geometry shader). This requires more than one render pass in order to generate a g-buffer (consisting of a normal buffer and optionally a depth buffer as well) to which we can apply discontinuity filters (eg. Sobel operator). Depth discontinuity is what enables suggestive contours and creases. My only quibble with this approach is the inability to provide finer control over inked edge widths, though with the right algorithm in the fragment shader, I'm sure this would be possible.

So the question now becomes more specific: How exactly would I go about getting variable edge widths, particularly on the outer silhouette, in a fragment shader?

Answer 1

I've decided to go with a fragment shader approach via discontinuity filtering of the depth buffer. Reasons for this are:

1. World vertex count is very, very high due to immense view distances, even with mesh LoD;
2. I am doing a number of other fragment shader operations, such as DoF blur which can benefit by the same structures (box or gaussian sampling/filtering) in the same pass.

Having tested it, I would say that in future projects I'd go for a geometry-based approach instead, for complexity reasons. The reason is that (as others suggested in the comments), fragment shader approaches to edge detection can be computationally intense, particularly with DoF implementations where the circle of confusion radius, and thus the number of samples per fragment, can be quite high. This is fortunately less of a concern for outline shaders.

Answer 2

There is actually a very easy general solution if you can use an after effect. The great thing is, it doesn't matter how high your poly-count is. Render a Depth map as greyscale, then create a dot in the line color you want, when the contrast between two adjacent pixels is higher than a threshold value. You can make the dot bigger according to the contrast and or light-level of the image you rendered.

I invented the toonShader algorithm back in 2000/2001, even before that French dude, who created this solution. Mine was based on actual materials geometry. Then there are basically two ways to do it: 1. look at normals, if normals two planes connected to a line are facing away and towards the camera, render that line, you can then use depth, lighting etc as cues for linedepths. 2. Look at rendered geomerty (so after perspective transformation) Take each linesegment, if the vertices of the connecting planes are on the same side of this linesegment, you render the line.. You can then do the same for line-thickness as in method 1. 3. You could make combinations of these three techniques, but I mentioned the first, because you indicated a huge poly-count.