# How to set the camera projection matrix so that its 4 rays always exactly hit the corners of a defined rectangle?

I have a camera, and I have a rectangle drawn in an arbitrary plane.

I need to change the projection matrix of the camera so that the rays along the edges of the camera's frustum always hit the corners of the rectangle.

The camera and the rectangle are supposed to be movable, rotatable, and the rectangle can be scaled.

Usage:

I am making a stereo-3D game with eye tracking, where my 2 eye vectors hit the exact same parallax plane.

That parallax plane is defined, and we have the corners of the display rectangle in that plane.

• What have you already tried and where have the problems begun? Commented Aug 31, 2019 at 11:24
• I think it's a good fit for math.stackexchange.com as well. But since you started bounty here, if it doesn't violate rules, you could start a similar question there but unrelated to Unity. And if you get a correct answer there - you could answer this question yourself with a more specifications related to gamedev and Unity, give a link to math.stackexchange answer which your answer would be based on. Or you could ask the person from math.stackexchange to answer this question here if they would like to have reputation on gamedev.stackexchange as well. Commented Aug 31, 2019 at 11:43
• mh... thanks - i think ill do it after this bounty expires Commented Aug 31, 2019 at 13:45
• Just a thought, you know your back plane points are defined in screenspace as (-1,-1, 1), (1, -1, 1), (1,1,1,), (-1, 1,1), if you invert your current projection matrix that would give you where the camera could be statically pointed to (defined by yourself as a static projection matrix) ,and the plane. Could you fashion World Matrix that then orientates world to the difference between your static projection and the parallax plane. Rather than move your camera, determine the difference between the camera backplane and your parallax plane? There should be no computational penalty. Commented Sep 2, 2019 at 23:48
• @OC_RaizW I hope this helps: gamedev.stackexchange.com/a/178411/106129. Commented Jan 16, 2020 at 10:59

What I understand is that you are trying to create a non-stereoscopic background to represent a very far view in the background in a stereoscopic view.

I would try creating a multi-camera setup instead of adjusting the main camera.

• Set the Display in Game view to match your 3D viewer target resolution. Create a new one if no option for stereoscopic views (e.g. 1080x1080).
• Create two additional cameras and two new layers for the background of each eye. (Background-LeftEye and RightEye).
• Set the Culling Mask of each camera to render only one of those layers.
• Set the Depth of each camera less than the Depth of your main camera.
• Target each camera only to one eye, left and right.
• Create two new Canvas for background for each eye.
• Set the Layer of each Canvas to your new layers respectively.
• Set Canvas Render Mode to Screen Space - Camera.
• Set Render Camera to your background cameras.
• Set Plane Distance to something that makes it work comfortably in your Scene view.
• Create an Image inside each Canvas and set its width and height to match your resolution (e.g. 1080x1080).
• Suggested use prefabs so that you don't need to setup everything twice.

I didn't build and test this on a VR viewer but you can use this as a starting point.

• It may be possible to achieve the same with a single background camera.
• I think you may instead use Render Texture instead of creating canvases.

As for the technique you ask, you can try raycasting and field of view instead of setting a projection matrix:

## Monoscopic view:

Assuming the rectangle is static from your last comment, you should be able to do it with raycasting on app start. I'll first propose a solution for monoscopic view, then extend to stereoscopic view.

• Set the display resolution canvas aspect ratio to the same as your rectangle.
• Make the rectangle child of the camera.
• Position the rectangle so that the camera is perpendicular to the rectangle and looking at its center (not applicable to stereoscopic view as both cameras will be non-perpendicular).
• Set a sufficiently large fieldOfView so that the whole rectangle is visible but not touching the edges.
• Start casting rays from screenspace to worldspace from the corners of your plane.
• Reduce fieldOfView until you get a raycast hit, preferably from all four corners.

## Stereoscopic view:

For that you need two additional background cameras similar to the Screen Space Camera solution, each with a rectangle as a child (preferably from the same prefab, otherwise cameras in the build will not be perpendicular to the rectangles.

Use a render texture if you want to modify the image rendered in each plane.

• Hi! I already solved this problem but it required me to do a weird matrix calculation. Commented Jun 7, 2022 at 1:04
• Here is the unity project. mega.nz/file/… If anyone wants to inspect what I have done - fix the issues and... Write a proper answer. Commented Jun 7, 2022 at 1:05