I currently output 3D scenes in Unity by rendering cube face textures from a scene camera, as you can see in the below Image:
From there, I convert the projection to 360^x180^ equirectangular projection, and map to the six projectors in the space based on a ground truth mesh which I use for calibration.
Therefore, I can display anything that the custom camera class views, including a plane mapped with a material connected to a Cg/HLSL shader.
My point was that I will see distortion in this case, since we are rendering a 2D plane rather than true depth.
Since shaders are entirely computed in screen space, and no mesh vertices are provided to the renderer. In our immersive environment, drawing a shader on a single quad stretched to the full projection extent will not produce the Z-depth perception we want. It is possible to remap the shader using the same cube face to equirectangular projections we use to adapt our 3D camera in Unity to the projector array (cube map to equirectangular: https://www.shadertoy.com/view/XsBSDR), however this approach is not compatible with our existing Unity camera (Omnity’s custom camera class) illustrated above.
Without our existing Omnity + Scalable projection pipeline, we could use a vertex + tessellation + geometry shader to tessellate and reproject all vertices into spherical coordinates. Emmanuel Durand created such a shader to generate a spherical stereographic rendering. This approach works but it has some drawbacks: many visual effects won't work once the vertices are transformed. For example, lighting and surface normal effects will most likely not be computed correctly.
So, the question is really, can we use GPU shaders with the above pipeline?