How to blend two cameras when traveling through a portal in Unity3D

Question

Before I get to my question, I know the most obvious solution would be to use the normalized view port rect, however I need more complex shapes than a rectangle, and I've looked in to using the view port rect and it seems to not be my solution.

EDIT: Some people were confused by my question, let me go in to a bit more detail. What's happening is as the player moves in to one portal, I create a clone FPS controller and move it out of the other. This gives me two cameras, and the view you see on the right above. It's just showing one camera, and clipping through the portal. What I want is something similar to this where the cameras blend to create the illusion of a smooth transition. What I want to do is delete everything from the green checker image to the left in the image below, and replace that with the other camera. That way you get the part of camera A's view that is peaking out of the portal, blended with the part of camera B's view that's peaking out of the other portal, to get one complete image. And as you move through the portal the cut changes appropriately.

I've been designing a portal system, I have everything down, including getting the player to smoothly move through the portal. My main problem now is getting that camera blend effect Valve does. I need two cameras to blend together seamlessly, as if you were poking your head through the portal. And it can't just be a rectangle, it has to match however the player is looking through the portal.

My best lead on this right now is to possibly project a depth mask shader behind each portal, then make the camera from the portal you're traveling to be depth only. Then somehow mix the two cameras. My main problem is figuring out exactly how I'd do this, how to make the second camera only render what's outside of the portal, and have the rest default to camera 1 to get one full screen projection.

If you could give me ideas, or explain how I may do this with the depth mask shader that would be a tremendous help. I'll continue to work on this and update as I make break throughs.

Answer 1

Understanding the Problem

From what I can see the problem you are describing is the result of the camera's near plane intersecting the plane defined by the portal. While this intersection occurs you can see behind the wall the portal is on.

This is similar to a problem experienced in other games when the player is just transitioning from above water to underwater. If the camera is just above the water, no post processing effect is applied to obfuscate the players view (make it dark and blurry and blue). So if the bottom of the near plane is just under the water the player can see clearly under water.

If I am correct, you can confirm it by changing the position of this plane when defining the projection matrix. As the distance from the camera origin to the near plane increases, so should the problem.

The Easy Solution

Making the near plane very close to the camera should almost eradicate this issue. This solution is not a complete one, but will yield a good enough result in the vast majority of cases and it's efficient.

The Full Solution

If just putting the near plane closer to the camera doesn't satisfy, then you can create a "mask" to blend the images generated by rendering the scene from the player and portal perspective.

Assuming you only allow for portals to be applied to flat surfaces, you can calculate the line of intersection between the camera's near plane and the plane defined by the portal (or the wall on which it lies). This line will split the screen into two parts. Determining which side of the line a screen pixel is on will let you know which render image to use, the portal image or the player camera image.

Keep in mind that if this problem is occurring then the camera view frustum has to be completely within the portal so the intersection line will always cut completely from one screen edge to another.

This link should help with the maths to find the line. The code below should be roughly correct.

The line of intersection is defined using a point on the line and the line direction. Below the intersection direction is calculated using the cross product of the portal normal and the camera view direction (near plane normal). A point on the line is given by casting a ray from a point on the near plane directly towards the portal plane (along the portal normal) and finding the intersection point.

Vector3 intersectionDir = Vector3.cross(portalNorm, viewDir);
Ray ray = new Ray(camPos + viewDir * camNearPlaneDist, portalNorm);
Vector3 intersectionPos = ray.intersects(new Plane(portalVert1, portalVert2, portalVert3);

Make sure the viewDir is a unit vector. portalVert1, 2 and 3 are just 3 of the 4 vertices used for the portal decal or the surface it's on. There are other ways to define the plane on which the portal lies, but I assume this is the most readily available information.

Once you have these two vectors to define the intersection line multiply each by the view and then projection matrices to get them in screen space.

You can then use a post process shader to blend these images. You select which image to use at each pixel by determining which side of the dividing line the current pixel lies on. This is done by taking the pixel position (which is also the position you use to look up the render target texel) and doing;

float d = (pixelX - intersectionPos.X) * intersectionDir.Y - (pixelY - intersectionPos.Y) * intersectionDir.X;

The side is given by whether d is more or less than 0. If it's exactly 0, then you're on the line.

For reference on the maths above see this.

This method can also be used when creating a depth mask / stencil buffer for use before rendering from the portal perspective. You can create a full screen quad and use the line to slice it.

Answer 2

The suggested answers were very good, but I ended up going for a different technique using a depth mask.

What you do is take THIS script and shader, you put the script on every object with a renderer in your scene and set the render queue to 3020 (I'll post a script to make this easy later).

Then, you create a box of planes (all facing inwards, in the picture you can't see the side of the box closest to you, but when you're inside the box all you should see is gray) behind BOTH of your portals like so: And set them on to a special layer of their own (I chose "DepthMask" for mine), then you add a material with the shader above to them.

You then take the main camera and uncheck your special layer from it's culling mask (I unchecked DepthMask layer) and set it's depth to 0.

Then when you're teleporting through and clone the camera, set the other camera's clear flags to "Depth Only" and it's depth to 1.

You then get a seamless screen blended between the two camera views.