Effective solution for showing the outline of the player sprite when behind objects or tiles that are meant to be in foreground

Question

I'm very new to Unity and I'm now experimenting with possible ways to show the player outline when he/she is behind objects or tiles. This is what I came up with:

My setup

• Built-in rendering pipeline
• Renderers sorted using based on Order in Layer. All the Tilemaps and the Player are at the same Z
• Transparency Custom Axis: (0, 1, 0). (All default sprite shaders are in the same Transparent Queue)
• Three Tilemaps: Bottom (grass, order in layer -2), Middle: (fence, order in layer -1), Top (house and player, order in layer 0)

My solution

• Create a Sprite GameObject (let's call it OutlineSprite) with the same sprite used by the player. Assign it as child of the PlayerSprite
• Assign, to the OutlineSprite, a "Order in Layer" greater than anything else.
• Assign, to the OutlineSprite, the same animator used for the Player sprite (and modify the c# script that sends notification to the animation controller whenever the player moves so that it sends notification to the OutlineSprite animation controller too)
• Assign, to the OutlineSprite, a material which uses a shader able to render the outline of a given sprite (I just copy/pasted a shader I found online which does more or less what I wanted, I'm not anywhere near to understand how to write a shader right now)
• Modify the player Sprite outline so that it exactly match the shape of the sprite
• Put a Sprite Mask in every place you need to create this effect (in the screenshot is just over the house roof)
• Configure the Player sprite to be "Visible outside mask" and the OutlineSprite to be "Visible inside mask"

This solution seems to do what I need but has many cons that I'd like to address, if possible.

Pro:

• simple solution that doesn't require extensive knowledge of advanced concepts

Cons:

• the player animation runs twice
• tedious to manually configure the outline of all the animation sprites
• tedious to manually place the sprite mask where needed

My question

Is there another (or multiple) solution to this problem that involve the least amount of changes and work (so that the setup fast and maintenance is minimal)? Ideally I'd like to only change the player's shader, if possible.

Answer 1

Following the "are you open to solutions that do swap the tile shader to write to the depth buffer?" suggestion by DMGregory on a comment in my question I implemented another possible solution which greatly simplifies the maintenance of the project.

Disclaimer

This solution is implemented using shaders. Saying that I'm new to shaders is an understatement. Take everything I say with caution, it might be wrong. Also, this is just a proof of concept and not a full implementation of a production ready shader. In fact the resulting shader will just paint the hidden part of a given color.

The idea

The idea is to write just one shader for the player so that when parts of the player sprite are behind another "object" (be it a gameobject or a tile), those part are visibile through the object and rendered differently than the rest of the sprite (as in the picture in my question, so to speak)

What we need

In a shader, to check if a certain pixel is in front, behind, or at the same level of a pixel already written in the frame buffer the ZBuffer is used. This means that, in the new PlayerShader, we have to make use of the ZTest to discriminate which pixel has to be written using the normal sprite assigned to the player or using the custom color. This immediately brought up the first problem: the unity official Sprite-Default shader (the default one used for sprites and tiles) has the ZWrite set to Off, which means that they do not write on the ZBuffer and so we cannot interrogate it and expecting a sounding answer about the pixel position. This is easily solved but requires a break in our initial idea: we have to modify the shader assigned to the tiles and not only the one assigned to the player.

But this is still not enough, and here lied one of the misconception I had. I initially thought that Z, in the ZBuffer, was the distance to the camera calculated using the sorting priorities already discussed. It took me a while to understand that the sorting priorities only deal with the order in which the renderer gets executed and this might create the effect that one element is on top of another, even tho they are at the very same distance from the camera. So, to make use of the ZBuffer, I think, objects have to be at different distance from the camera.

Changing in the setup

In order to implement the player shader, I had to make a few modifications in my setup:

• Create a new material with a news shader for the tiles. The new shader is simply a copy-paste of the official Sprite-Default.shader with ZWrite On instead of Off
• Modify the Z for the Top tielemap (the one containing the house, but in general for all objects/tiles that need to be on top of the player) to a number greater than the Z of the player (but less than the camera)

Player Shader

As I said: this is just a proof of concept, far from working in every situation. The important parts are:

• The queue is Transparent+1 so that it gets rendered after all other sprites. At this point the Top tilemaps with the house should have already written in the ZBuffer the information on its position.

• The first pass is again the same copy-paste of the default unity sprite shader, with the exception that we only execute it when ZTest LEqual

• The second pass is executed when ZTest Greater instead and it's its job to paint the outline of the player behind the object. In this case we just color it without doing anything fancy.

    Shader "Test/PlayerShader"
    {
      Properties
      {
        _OutlineColor ("Outline Color", Color) = (0,0,1,1)
        [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
        _Color ("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
      }
      SubShader
      {
        Tags
        { 
            "Queue"="Transparent+1" 
            "IgnoreProjector"="True" 
            "RenderType"="Transparent" 
            "PreviewType"="Plane"
            "CanUseSpriteAtlas"="True"
        }
  
        Cull Off
        Lighting Off
        ZWrite On
        Blend One OneMinusSrcAlpha
  
        Pass
        {
            Name "ForeGround"
            ZTest LEqual
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile _ PIXELSNAP_ON
            #include "UnityCG.cginc"
  
            struct appdata_t
            {
                float4 vertex   : POSITION;
                float4 color    : COLOR;
                float2 texcoord : TEXCOORD0;
            };
  
            struct v2f
            {
                float4 vertex   : SV_POSITION;
                fixed4 color    : COLOR;
                float2 texcoord  : TEXCOORD0;
            };
  
            fixed4 _Color;
  
            v2f vert(appdata_t IN)
            {
                v2f OUT;
                OUT.vertex = UnityObjectToClipPos(IN.vertex);
                OUT.texcoord = IN.texcoord;
                OUT.color = IN.color * _Color;
                #ifdef PIXELSNAP_ON
                OUT.vertex = UnityPixelSnap (OUT.vertex);
                #endif
  
                return OUT;
            }
  
            sampler2D _MainTex;
            sampler2D _AlphaTex;
            float _AlphaSplitEnabled;
  
            fixed4 SampleSpriteTexture (float2 uv)
            {
                fixed4 color = tex2D (_MainTex, uv);
  
    #if UNITY_TEXTURE_ALPHASPLIT_ALLOWED
                if (_AlphaSplitEnabled)
                    color.a = tex2D (_AlphaTex, uv).r;
    #endif //UNITY_TEXTURE_ALPHASPLIT_ALLOWED
  
                return color;
            }
  
            fixed4 frag(v2f IN) : SV_Target
            {
                fixed4 c = SampleSpriteTexture (IN.texcoord) * IN.color;
                c.rgb *= c.a;
                return c;
            }
        ENDCG
        }
  
        Pass {
            Name "Background"
            ZTest Greater
            Blend SrcAlpha OneMinusSrcAlpha 
  
            CGPROGRAM 
  
            #pragma vertex vert 
            #pragma fragment frag
  
            float4 vert(float4 vertexPos : POSITION) : SV_POSITION 
            {
            return UnityObjectToClipPos(vertexPos);
            }
  
            float4 _OutlineColor;
  
            float4 frag(void) : Color 
            {
                return _OutlineColor;
            }
  
            ENDCG  
        }
    }
  }