How would I mask the player node out of a canvas shader?

Question

I'm making a 3D game, and I want to create a post processing effect using a canvasitem and a colorrect, that effects everything on screen apart from the player. I already have the shader itself made, but my first attempt at masking the player out didn't work. I used 2 cameras, one for the world and one for the player, but that meant the player was in front of everything in the world.

Ideally, I need to be able to make a texture like the one below (forgive the MS Paint mockup) to use for transparency in the canvas shader. How would I do this?

Answer 1

You will setup a SubViewport (you would not need a container for it, since - I presume - it would not be shown), make sure its render mode and its clear mode are set to always. And the SubViewport can have its own world (I'll get back to this).

Inside the SubViewport you want to have a different Camera3D, and (maybe) a copy of the character model with a material override (I'll get back to this). The material override must be an all white material, unshaded (and you can also have depth test disabled).

You need the clear color to be black. If the clear color in project settings is black, you are done. Otherwise, you can also set a WorldEnviroment inside the SubViewport and set its background to custom color and pick black (this only works if the SubViewport has its own world). I suppose that as last resort you could have black quad match the view (which you can do by giving it a material that moves the vertex), and either have it match the far plane, or have the material override for the character has depth test disable.

You would - by script - keep the Camera3D and character inside the SubViewport synchronized. Copy their global_transform in _process, and any other relevant property (e.g. field of view of the camera). I suppose animations might be a challenge, depending how you are handling them for your character.

Could you use the same character instead of a copy? Sure, if you do not set the SubViewport to have its own world, it will default to the same as the parent. However, since you need different materials, you will have to duplicate the meshes. To make one copy of the mesh show in the SubViewport and another outside, give them different layers (it is under VisualInstance3D), and then give matching cull masks to the Camera3Ds.

Then you can get the texture from the SubViewport.If you need to set it in the Inspector (for example as a parameter of your canvas shader), pick ViewportTexture and point it to the SubViewport. Note: The SubViewport needs to be first in the scene tree (before where you use the ViewportTexture) so that during initialization the SubViewport is created before the ViewportTexture that references it. Alternatively, could also take a texture from the SubViewport from a script.

Answer 2

I have found an answer to how to do this. Other methods I have found, would single the player out of the environment, and make it show in front of everything.

For my method, I make a subViewport, with a second camera, and make it copy the transform of the main camera. I put the player on a second cull layer, and have the second camera not see the player. I then make a shader which compares the view of the main camera, and the camera without the player, and do a difference operation (abs(a-b)) to find what is different between the 2 camera views, which would be everywhere the player is in the image. This also takes obstacles into account, as the obstacles would be the same in the 2 views.

The only times this method wouldn't work is if the player happens to be standing in front of an object with the exact same RGB value as it, which for my game, would never happen.