Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I see a few broad approaches out there to doing cel shading:

  1. Duplication & enlargement of model with flipped normals (not an option for me)
  2. Sobel filter / fragment shader approaches to edge detection
  3. Stencil buffer approaches to edge detection
  4. Geometry (or vertex) shader approaches that calculate face and edge normals

Am I correct in assuming the geometry-centric approach gives the greatest amount of control over lighting and line thickness, as well eg. for terrain where you might see the silhouette line of a hill merging gradually into a plain?

What if I didn't need pixel lighting on my terrain surfaces? (And I probably won't as I plan to use cell-based vertex- or texturemap-based lighting/shadowing.) Would I then be better off sticking with the geometry-type approach, or go for a screen space / fragment approach instead to keep things simpler? If so, how would I get the "inking" of hills within the mesh silhouette, rather than only the outline of the entire mesh (with no "ink" details inside that outline? (AKA suggestive contours, creases).

Lastly, is it possible to cheaply emulate the flipped-normals approach, using a geometry shader? My concern with this is that I could certainly duplicate every single vertex and scale these accordingly, but how would I approach flipping normals, and distinct colouring in the fragment shader?

What I want - varying line thickness with intrusive lines inside the silhouette...

enter image description here

What I don't want...

enter image description here


EDIT: Further research has turned up the following...

Since I have a massive vertex count on terrain, even considering distance-based LoD, neither flipped-normals nor a geometry shader based approach (even with frustum culling) would be a sensible option due to sheer computational complexity involved in duplication & scaling of all uploaded vertices.

Considering that I don't need per-pixel lighting in the form of solid-tone shading on the terrain surfaces, it also becomes less prudent to consider any face-normal based approaches -- otherwise a requirement for correct surface lighting -- as these are naturally quite expensive to calculate. It is however true that they give the finest degree of control; for example, the ability to shade edges using "artistic" strokes: Beautiful, but again, not really viable for a massively complex game environment.

Stencil buffers are something I'd rather avoid as I'd prefer to do all work in shaders. (The above example with the red outline was done with a stencil buffer -- old school.)

This leaves fragment shader image-space approaches. Computational complexity is reduced to the number of fragments rather than the number of vertices (in my case, this is 10-100x fewer operations than I would have to do in the geometry shader). This requires more than one render pass in order to generate a g-buffer (consisting of a normal buffer and optionally a depth buffer as well) to which we can apply discontinuity filters (eg. Sobel operator). Depth discontinuity is what enables suggestive contours and creases. My only quibble with this approach is the inability to provide finer control over inked edge widths, though with the right algorithm in the fragment shader, I'm sure this would be possible.

So the question now becomes more specific: How exactly would I go about getting variable edge widths, particularly on the outer silhouette, in a fragment shader?

share|improve this question
    
Varying line thickness is a feature of the geometric approach. Can't efficiently get it any other way. Since you want that, I don't see pixel shaders as being the "more efficient method": number of pixels to be searched = (line_thickness * 2 + 1) ^ 2 - 1. That would basically mean that your post-process shader will be a lot (guesstimate: 10x) slower if max. line width equals 2. Just drop pre-judging and try the geometry shader approach. –  snake5 Dec 4 '12 at 14:07
    
@snake5 You're honestly suggesting that I run through 150 million vertices per render update (about 25 million with view culling), in the GS? I'm not really buying that, but thanks for the input. Refer back to what I stated above about vertex counts and complexity. Even with your 10x figure, the fragment shader would at least draw even, and probably do a hell of a lot better. –  Nick Wiggill Dec 4 '12 at 14:26
    
25 million vertices (linear reads/writes) vs. 23 million texture samples (rather poorly cached memory reads + decompression of render target data) @ 1280x720. Limiting/reducing vertex count is a lot easier, by the way. Especially these days, when big monitors (1920x1080+) & multimonitor setups are getting very popular. –  snake5 Dec 4 '12 at 15:02
    
what about using same algorithm as the one generating red outlines, and then thinning/growing lines based on their depth? –  Ali.S Dec 4 '12 at 18:57
    
@Gajoo, As I was saying in the question, it's not just about the silhouette. Note the intrusive lines behind the girl's shoulder in the first image. Those are a function of the flipped-normals approach, and act as suggestive contours / creases. I need same. –  Nick Wiggill Dec 4 '12 at 19:03
add comment

1 Answer 1

up vote 2 down vote accepted

I've decided to go with a fragment shader approach via discontinuity filtering of the depth buffer. Reasons for this are:

  1. World vertex count is very, very high due to immense view distances, even with mesh LoD;
  2. I am doing a number of other fragment shader operations, such as DoF blur which can benefit by the same structures (box or gaussian sampling/filtering) in the same pass.

Having tested it, I would say that in future projects I'd go for a geometry-based approach instead, for complexity reasons. The reason is that (as others suggested in the comments), fragment shader approaches to edge detection can be computationally intense, particularly with DoF implementations where the circle of confusion radius, and thus the number of samples per fragment, can be quite high. This is fortunately less of a concern for outline shaders.

share|improve this answer
    
I think going with normals discontinuity would give you better results in case you got flat normals. You can vary the line width by taking samples farther apart. –  Grieverheart Dec 7 '12 at 11:35
    
@Grieverheart This is the accepted answer, because the method I'm using works just fine. Thanks. –  Nick Wiggill Dec 7 '12 at 11:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.