How can I draw outlines around 3D models? I'm referring to something like the effects in a recent Pokemon game, which appear to have a single-pixel outline around them:
This effect is particularly common in games that make use of cel shading effects, but is actually something that can be applied independently of the cel shading style.
What you are describing is called "feature edge rendering," and is in general process of highlighting the various contours and outlines of a model. There are many techniques available and many papers on the subject.
A simple technique is to render only the silhouette edge, the outmost outline. This can be done as simply as rendering the original model with a stencil write, and then rendering it again in thick wireframe mode, only where there was no stencil value. See here for an example implementation.
That will not highlight the interior contour and crease edges, though (as shown in your pictures). Generally, to do that effectively, you need to extract information about the mesh's edges (based on discontinuities in the face normals on either side of the edge, and build up a data structure representing each edge.
You can then write shaders to extrude or otherwise render those edges as regular geometry overtop your base model (or in conjunction with it). The position of an edge, and normals of the adjacent faces relative to the view vector, are used to determine if a specific edge can be drawn.
You can find further discussion, details and papers with various examples on the internet. For example:
I don't think any of the other answers here will achieve the effect in Pokémon X/Y. I can't know exactly how it's done, but I figured out a way which seems like pretty much what they do in the game.
In Pokémon X/Y, outlines are drawn both around the silhouette edges and on other non-silhouette edges (like where Raichu's ears meet his head in the following screenshot).
Looking at Raichu's mesh in Blender, you can see the ear (highlighted in orange above) is just a separate, disconnected object which intersects the head, creating an abrupt change in the surface normals.
Based on that, I tried generating the outline based on the normals, which requires rendering in two passes:
First pass: Render the model (textured and cel-shaded) without the outlines, and render the camera-space normals to a second render target.
Second pass: Do a full-screen edge detection filter over the normals from the first pass.
The first two images below show the outputs of the first pass. The third is the outline by itself, and the last is the final combined result.
Here's the OpenGL fragment shader I used for edge detection in the second pass. It's the best I was able to come up with, but there might be a better way. It's probably not very well optimized either.
And before rendering the first pass I clear the normals' render target to a vector facing away from the camera:
I read somewhere (I'll put a link in the comments) that the Nintendo 3DS uses a fixed-function pipeline instead of shaders, so I guess this can't be the way it's done in the game exactly, but for now I'm convinced my method is close enough.
For smooth models (very important), this effect is fairly simple. In your fragment/pixel shader you will need the normal of the fragment being shaded. If it is very close to perpendicular (
This approach "consumes" a little bit of the model to do the outline. This may or may not be what you want. It's very difficult to tell from the Pokemon picture if this is what is being done. It depends on if you expect the outline to be included in any silhouette of the character or if you'd rather have the outline enclose the silhouette (which requires a different technique).
The highlight will be on any part of the surface where it transitions from front-facing to back-facing, including "inner edges" (like the legs on the green Pokemon, or its head - some other techniques would not add any outline to those).
Objects that have hard, non-smooth edges (like a cube) will not receive a highlight in the desired locations with this approach. That means this approach is not an option at all in some cases; I have no idea if Pokemon models are all smooth or not.
The simplest way to do this, common on older hardware before pixel/fragment shaders, and still used on mobile, is to duplicate the model, reverse the vertex winding order so that the model displays inside out (or if you want to, you can do this in your 3D asset creation tool, say Blender, by flipping surface normals -- same thing), then expand the entire duplicate slightly around it's centre, and finally colour/texture this duplicate completely black. This results in outlines around your original model, if it's a simple model such as a cube. For more complex models with concave forms (such as that in the image below), it is necessary to manually tweak the duplicate model to be somewhat "fatter" than its original counterpart, like a Minkowski Sum in 3D. You could start by pushing each vertex out a bit along its normal to form the outline mesh, as Blender's Shrink/Fatten transform does.
Screen space / pixel shader approaches tend to be slower and harder to implement well, but OTOH don't double the number of vertices in your world. So if you're doing high poly work, best opt for that approach. Given modern console and desktop capacity for processing geometry, I'd not worry about a factor of 2 at all. Cartoon-style = low poly for sure, thus duplicating geometry is easiest.
You can test the effect for yourself in e.g. Blender without touching any code. Outlines should look like the image below, note how some are internal, e.g. under arm. More detail here.
The most common way I've seen this done is via a second render pass on your model. Essentially, duplicate it and flip the normals, and shove that into a vertex shader. In the shader, scale each vertex along its normal. In the pixel/fragment shader, draw black. That'll give you both external and internal outlines, like around lips, eyes, etc. This is actually a fairly cheap draw call, if nothing else it's generally cheaper than post processing the line, depending on the number of models and their complexity. Guilty Gear Xrd uses this method because it's easy to control the thickness of the line via vertex color.
The second way of doing inner lines I learned from the same game. In your UV map, align your texture along the u or v axis, particularly in areas where you want an inner line. Draw a black line along either axis, and move your UV coordinates into or out of that line to create the inner line.
See the video from GDC for a better explanation: https://www.youtube.com/watch?v=yhGjCzxJV3E