How to set the camera projection matrix so that its 4 rays always exactly hit the corners of a defined rectangle?

Question

I have a camera, and I have a rectangle drawn in an arbitrary plane.

I need to change the projection matrix of the camera so that the rays along the edges of the camera's frustum always hit the corners of the rectangle.

The camera and the rectangle are supposed to be movable, rotatable, and the rectangle can be scaled.

Usage:

I am making a stereo-3D game with eye tracking, where my 2 eye vectors hit the exact same parallax plane.

That parallax plane is defined, and we have the corners of the display rectangle in that plane.

Answer 1

What I understand is that you are trying to create a non-stereoscopic background to represent a very far view in the background in a stereoscopic view.

I would try creating a multi-camera setup instead of adjusting the main camera.

• Set the Display in Game view to match your 3D viewer target resolution. Create a new one if no option for stereoscopic views (e.g. 1080x1080).
• Create two additional cameras and two new layers for the background of each eye. (Background-LeftEye and RightEye).
• Set the Culling Mask of each camera to render only one of those layers.
• Set the Depth of each camera less than the Depth of your main camera.
• Target each camera only to one eye, left and right.
• Create two new Canvas for background for each eye.
• Set the Layer of each Canvas to your new layers respectively.
• Set Canvas Render Mode to Screen Space - Camera.
• Set Render Camera to your background cameras.
• Set Plane Distance to something that makes it work comfortably in your Scene view.
• Create an Image inside each Canvas and set its width and height to match your resolution (e.g. 1080x1080).
• Suggested use prefabs so that you don't need to setup everything twice.

I didn't build and test this on a VR viewer but you can use this as a starting point.

• It may be possible to achieve the same with a single background camera.
• I think you may instead use Render Texture instead of creating canvases.

As for the technique you ask, you can try raycasting and field of view instead of setting a projection matrix:

Monoscopic view:

Assuming the rectangle is static from your last comment, you should be able to do it with raycasting on app start. I'll first propose a solution for monoscopic view, then extend to stereoscopic view.

• Set the display resolution canvas aspect ratio to the same as your rectangle.
• Make the rectangle child of the camera.
• Position the rectangle so that the camera is perpendicular to the rectangle and looking at its center (not applicable to stereoscopic view as both cameras will be non-perpendicular).
• Set a sufficiently large fieldOfView so that the whole rectangle is visible but not touching the edges.
• Start casting rays from screenspace to worldspace from the corners of your plane.
• Reduce fieldOfView until you get a raycast hit, preferably from all four corners.

Stereoscopic view:

For that you need two additional background cameras similar to the Screen Space Camera solution, each with a rectangle as a child (preferably from the same prefab, otherwise cameras in the build will not be perpendicular to the rectangles.

Use a render texture if you want to modify the image rendered in each plane.