What I understand is that you are trying to create a non-stereoscopic background to represent a very far view in the background in a stereoscopic view.
I would try creating a multi-camera setup instead of adjusting the main camera.
- Set the Display in Game view to match your 3D viewer target resolution. Create a new one if no option for stereoscopic views (e.g. 1080x1080).
- Create two additional cameras and two new layers for the background of each eye. (Background-LeftEye and RightEye).
- Set the
Culling Mask of each camera to render only one of those layers.
- Set the
Depth of each camera less than the Depth of your main camera.
- Target each camera only to one eye, left and right.
- Create two new
Canvas for background for each eye.
- Set the
Layer of each Canvas to your new layers respectively.
Canvas Render Mode to
Screen Space - Camera.
Render Camera to your background cameras.
Plane Distance to something that makes it work comfortably in your Scene view.
- Create an
Image inside each Canvas and set its width and height to match your resolution (e.g. 1080x1080).
- Suggested use prefabs so that you don't need to setup everything twice.
I didn't build and test this on a VR viewer but you can use this as a starting point.
- It may be possible to achieve the same with a single background camera.
- I think you may instead use
Render Texture instead of creating canvases.
As for the technique you ask, you can try raycasting and field of view instead of setting a projection matrix:
Assuming the rectangle is static from your last comment, you should be able to do it with raycasting on app start. I'll first propose a solution for monoscopic view, then extend to stereoscopic view.
- Set the display resolution canvas aspect ratio to the same as your rectangle.
- Make the rectangle child of the camera.
- Position the rectangle so that the camera is perpendicular to the rectangle and looking at its center (not applicable to stereoscopic view as both cameras will be non-perpendicular).
- Set a sufficiently large
fieldOfView so that the whole rectangle is visible but not touching the edges.
- Start casting rays from screenspace to worldspace from the corners of your plane.
fieldOfView until you get a raycast hit, preferably from all four corners.
For that you need two additional background cameras similar to the Screen Space Camera solution, each with a rectangle as a child (preferably from the same prefab, otherwise cameras in the build will not be perpendicular to the rectangles.
Use a render texture if you want to modify the image rendered in each plane.