The question is pretty much in the title. Are there notable 3D projections other than orthographic and perspective that could be used in a 3D system such as OpenGL?
In particular, are there any that have a particular use in game design?
Yes, To name a few:
The Pannini projection, for example, can capture wide fields of view in nice ways. (totally just my opinion)
I think implementation details would be beyond the scope of this specific question.
EDIT: Thanks for the comment, I did misspell Pannini. And to make this edit worthwhile here are a few more:
It depends on what you mean by "that could be used in a 3D system such as OpenGL". :)
Narrowly speaking, 3D graphics hardware and APIs like OpenGL only deal correctly with linear projections - projections that map straight lines in world space to straight lines on the image. They never distort something into a curved shape (unless it was curved to begin with). This is because GPUs depend on triangles having straight edges in order to rasterize them correctly on screen.
If we limit ourselves to linear projections, there aren't too many choices. In addition to the standard ortho and perspective, there are also "off-center" variations which are obtained by shearing the view frustum.
These projections can be represented with the usual 4×4 projection matrix and used in a 3D API without problems.
Then there are the nonlinear projections, which don't have the restriction of mapping straight lines to straight lines; they're allowed to distort things into curves. There are lots of these, including cylindrical, spherical, various types of fisheye, and others.
With nonlinear projections, you can't simply use a projection matrix; you have to somehow implement the projection yourself using shaders. One way is to write a custom vertex shader to do the projection per vertex. The GPU will still draw the triangle with straight lines between the vertices, so while small triangles will be pretty close to correct, larger triangles will be noticeably wrong. This can cause problems, and geometry may need to be subdivided very finely in order to render correctly.
A second approach to getting a nonlinear projection is to render the scene using a regular perspective projection first (maybe to a cubemap, which has a perspective projection for each cube face), then apply a post-process pixel shader to resample it to the desired nonlinear projection. This has the advantage of not requiring geometry modifications, but it can be more expensive due to the extra rendering work required, and the result can be blurry in some areas if the first-pass render doesn't have enough resolution. The post-process approach is used by Oculus Rift games, for example, to implement the nonlinear projection the Rift requires to work with its lenses.
I can't believe it but nobody has mentioned Isometric Projection which used to be quite common.