In its original incarnation, a stencil buffer was a one-bit-per-pixel (i.e. black or white, but no grays) framebuffer. You could render to it whatever you wanted like any other framebuffer. Then, later, you could use the contents of that buffer to "stencil" or mask out when drawing to your regular buffer.
An example: Let's say you're making a driving game. You want to have a little rear-view mirror onscreen that shows you what's behind the car. You'll need to render a view pointing behind the car, but you only want to render that within the little rounded rectangle of the rear-view mirror. The typical solution is:
- Render the rounded rectangle shape to the stencil buffer.
- Enable stencilling.
- Render the backwards pointing view onto the regular buffer.
The stencil will then mask it out so that you only draw into the shape of the mirror.
Now that render pipelines are much more flexible and programmable, stencil buffers are used as just a generic 1-bit framebuffer that you can do whatever you want with. Shadows are a common use case.