We can do this by making a custom version of the standard sprite shader (so it still does everything the one you're used to can do), and modifying its blending operation.
Start by going to the Unity Download Archive, and grab the built-in shaders for the version of Unity you're using. I'm currently on 2020.3.7f1 (LTS)
Inside that zip, you'll want to copy DefaultResourcesExtra/Sprites-default.shader
- place the copy in your project's Assets
folder with a new name, like "Sprites-invert.shader"
Now open up the text file and make two modifications:
First, at the very top of the file, change the shader's name to match the name you picked for the file, so it doesn't collide with the built-in shader.
Shader "Sprites/Invert"
{
Then scroll down to this line:
Blend One OneMinusSrcAlpha
This says "When blending this sprite with the background, the output colour should be the colour of this sprite times 'one', plus the colour already in the frame times 'one minus the alpha value of this sprite'". That gives us what's called premultiplied alpha blending.
We'll change it to this:
Blend OneMinusDstColor OneMinusSrcColor
Now we're multiplying the foreground colour by the inverse of the background, and the background by the inverse of the foreground. That gives us...
Foreground |
Background |
First Term |
+ |
Second Term |
= |
Output |
1 (White) |
0 (Black) |
(1) * (1-0) = 1 |
+ |
(0) * (1-1) = 0 |
= |
1 (White) |
1 (White) |
1 (White) |
(1) * (1-1) = 0 |
+ |
(1) * (1-1) = 0 |
= |
0 (Black) |
0 (Black) |
0 (Black) |
(0) * (1-0) = 0 |
+ |
(0) * (1-0) = 0 |
= |
0 (Black) |
0 (Black) |
1 (White) |
(0) * (1-1) = 0 |
+ |
(1) * (1-0) = 1 |
= |
1 (White) |
...And for values in between we get greys, so you don't lose your antialiased adges like on that triangular sprite in your mask example.
Make a material that uses your new shader, and assign it to the SpriteRenderer components for your foreground sprites. You'll get a result something like this:
(Using this "Sauropod skeleton" icon by Caro Asercion as an example)
You can apply a post effect at the end to remap your colours if you want the white and black to look like something else. This keeps the shader simpler than trying to use those special colour values throughout the pipeline.
What's nice about this is that it has exactly the same cost as the original sprite shader - we didn't add even a single new instruction, just changed the parameters of the blending operation we were already doing anyway That's much cheaper than mask-based solutions that require extra rendering/sampling from the mask texture.
If you have both sprites using the default material and sprites using this new material in your scene, you'll pay a little cost due to splitting them into separate draw calls for each shader. But if you're using this new sprite shader exclusively then it's effectively free. 😁