I am writing a game that very much relies on real-time shadowing and would like to implement shadow volumes. I was thinking of doing shadow mapping, but the platform I am using is Molehill/Stage3D and it's limited to 8 bits per channel, BGRA textures, and shaders can only read from textures, not depth buffers. I have never done stencil shadows before, but would it be possible to generate a shadow volume entirely on the GPU without the use of Geometry shaders, and only vertex shaders?
Is there any way of pretending that a depth buffer is a BGRA texture (by creating a texture that overlaps the depth buffer in memory, maybe), or will the API not allow that? If it will, you can decode the depth value in the pixel shader by working out which three channels contain the depth value, which ones are the high/middle/low bytes of the value, and doing something like:
float depth = (highByte * (255.0 * 65536.0) + middleByte * (255.0 * 256.0) + lowByte * 255.0) / 16777215.0
(Here, I'm assuming
lowByte are in the 0-1 range, and
depth also comes out in the 0-1 range.)
Anyway, if that's not an option, you can certainly do stencil shadows - they were invented a long time before geometry shaders existed. However, you will either have to do some expensive CPU work to generate the silhouette edges and project them, OR preprocess the mesh to add additional vertices that can then be projected by a vertex shader. See this article for the details (in the section "Shadow Volumes Powered By Vertex Shaders").