How to render to texture with VR distortion?

Question

I've also asked the question on Unity Answers.

So, I want to do some post processing effects in VR. I put a camera as a child of the Main Camera and set it to render-to-texture. Then, I take what is rendered, and display it (with my post-proc shader) on a quad in front of the Main Camera (The main camera and the child camera render different layers, so I don't render the same thing twice). (Also worth noting- I know that the depth will be collapsed out of the rendered-to-texture content; this is intentional.) The problem is, in this process, the positional data (that was rendered to texture) gets misaligned.

I've already fixed a problem where I have to manually set

camera_project.GetComponent<Camera>().aspect
 = main_camera.GetComponent<Camera>().aspect; 

every frame. (Changing/setting the screen size auto sets the aspect ratio of the main camera- it does not automatically set the aspect ratio of the child camera).

So, with just that, everything lines up great and looks fine when it isn't in VR. But, when it is in VR, it appears that maybe some distortion is happening to the main camera that isn't happening to the render-to-texture camera, and thus, the overlay (the post-processed content) isn't lining up.

So what can I do to emulate the VR distortion in the child camera? Are there other properties of the Main Camera that are getting auto-set in VR that I can manually copy to the Main Camera?

Answer 1

The best I've been able to come up with (not 100% perfect) is to just set the FOV of the render-to-texture cameras to be 96.5 (the main camera is set to 60). I just came up with this number experimentally. I'm not sure if under the hood it's just auto-setting the fov of the main to something similar to 96.5, but I can't find any hard info...

edit: Ok! I found hard info. Looks like it just auto-sets the fov of the main camera during runtime to exactly 96.01604. So, either set your render-to-texture cameras to that, or assign it at runtime from the main camera.

Answer 2

I have searched a lot on the internet. I can save you time. One day as of this post Unity offers absolutely nothing to get a single render texture with stereoscopic view using Single Instance.

But of course, there is the issue that if you want to make a relatively complex game Single Instance is practically mandatory.

The only solution I have found is also quite simple: use two cameras for each of the eyes of the HMD viewer. So, for each stereoscopic texture that we want, for example a portal, we will have to create two cameras that will track the position of the eyes in a relative way, recording the texture in two render textures that we can assign to each of the textures of a shader that unify in a single vision.