I see a few broad approaches out there to doing cel shading:
- Duplication & enlargement of model with flipped normals (not an option for me)
- Sobel filter / fragment shader approaches to edge detection
- Stencil buffer approaches to edge detection
- Geometry (or vertex) shader approaches that calculate face and edge normals
Am I correct in assuming the geometry-centric approach gives the greatest amount of control over lighting and line thickness, as well eg. for terrain where you might see the silhouette line of a hill merging gradually into a plain?
What if I didn't need pixel lighting on my terrain surfaces? (And I probably won't as I plan to use cell-based vertex- or texturemap-based lighting/shadowing.) Would I then be better off sticking with the geometry-type approach, or go for a screen space / fragment approach instead to keep things simpler? If so, how would I get the "inking" of hills within the mesh silhouette, rather than only the outline of the entire mesh (with no "ink" details inside that outline? (AKA suggestive contours, creases).
Lastly, is it possible to cheaply emulate the flipped-normals approach, using a geometry shader? My concern with this is that I could certainly duplicate every single vertex and scale these accordingly, but how would I approach flipping normals, and distinct colouring in the fragment shader?
What I want - varying line thickness with intrusive lines inside the silhouette...
What I don't want...
EDIT: Further research has turned up the following...
I'm going to answer this, but I'm not going to accept my answer as I would still like to hear from others... this is more of an update as opposed to a firm answer, for now.
Since I have a massive vertex count on terrain, even considering distance-based LoD, neither flipped-normals nor a geometry shader based approach (even with frustum culling) would be a sensible option due to sheer computational complexity involved in duplication & scaling of all uploaded vertices.
Considering that I don't need per-pixel lighting in the form of solid-tone shading on the terrain surfaces, it also becomes less prudent to consider any face-normal based approaches -- otherwise a requirement for correct surface lighting -- as these are naturally quite expensive to calculate. It is however true that they give the finest degree of control; for example, the ability to shade edges using "artistic" strokes: Beautiful, but again, not really viable for a massively complex game environment.
Stencil buffers are something I'd rather avoid as I'd prefer to do all work in shaders. (The above example with the red outline was done with a stencil buffer -- old school.)
This leaves fragment shader image-space approaches. Computational complexity is reduced to the number of fragments rather than the number of vertices (in my case, this is 10-100x fewer operations than I would have to do in the geometry shader). This requires more than one render pass in order to generate a g-buffer (consisting of a normal buffer and optionally a depth buffer as well) to which we can apply discontinuity filters (eg. Sobel operator). Depth discontinuity is what enables suggestive contours and creases. My only quibble with this approach is the inability to provide finer control over inked edge widths, though with the right algorithm in the fragment shader, I'm sure this would be possible.
So the question now becomes more specific: How exactly would I go about getting variable edge widths, particularly on the outer silhouette, in a fragment shader?