GLSL 2D Silhouette

Question

I'm trying to render a silhouette of any entity behind foreground objects to indicate their position to the player. Here's an example of what I want to achieve, from Hyper Light Drifter:

From what I've read online about GLSL and the ways I might be able to achieve this, the closest thing I can think of to achieve this would be to use the stencil buffer, which was also indicated in this question:

2D Hidden Object Silhouette Shader

I don't quite understand how I can achieve this within LibGDX. Should I be using a vertex / fragment shader, or the native Gdx methods? This is my current rendering workflow:

Render Lower Tiled Map Layers
Render Entities
Render Upper Tiled Map Layers

From what I understand, I would have to:

1. Write the top layer to the stencil buffer
2. Render the entities (silhouettes) within the stencil

Is this right? If anyone can enlighten me, it would be greatly appreciated!

Answer 1

All of the drawing code is separated into their own systems responsible for rendering what is on the screen. Differentiate between Lower and Upper layers by naming the layer you want the Entities to be on. I use a custom interface designed to separate the render method of LibGDX into update(delta) and draw(SpriteBatch, delta), similar to C#'s XNA as this was more familiar for me.

You need two fragment shaders and one vertex shader to get the desired result. The vertex shader is a simple passthrough shader:

    attribute vec4 a_color;
    attribute vec4 a_position;
    attribute vec2 a_texCoord0;
    
    uniform mat4 u_projTrans;
    
    varying vec4 v_color;
    varying vec2 v_texCoord0;
    
    void main() {
        v_color = a_color;
        v_color.a = v_color.a * (255.0/254.0);
        v_texCoord0 = a_texCoord0;
    
        gl_Position = u_projTrans * a_position;
    }

Then you need a fragment shader to discard any pixels from the upper layers with an alpha value of 0 from the Stencil Buffer:

    varying vec4 v_color;
    varying vec2 v_texCoord0;
    
    uniform sampler2D u_texture;
    
    void main() {
        gl_FragColor = v_color * texture2D(u_texture, v_texCoord0);
    
        if (gl_FragColor.a == 0.0) {
            discard;
        }
    }

And finally, another fragment shader to render entity silhouettes:

    varying vec4 v_color;
    varying vec2 v_texCoord0;
    
    uniform sampler2D u_texture;
    
    void main() {
        gl_FragColor = vec4(0.0, 0.75, 1.0, 0.25) * texture2D(u_texture, v_texCoord0).a;
    }

1. Render Lower Tiled Map Layers.

    public class LowerTiledMapRenderSystem extends EntitySystem implements Drawable {
   
        private int[] layerIndeces;
   
        private static final OrthogonalTiledMapRenderer renderer = LevelManager.RENDERER;
   
        public LowerTiledMapRenderSystem() {
            List<Integer> layers = new LinkedList<>();
   
            for (int i = 0; i < renderer.getMap().getLayers().getCount(); i++) {
                if (renderer.getMap().getLayers().get(i).getName().equalsIgnoreCase("entities")) {
                    layerIndeces = layers.stream().mapToInt(Integer::intValue).toArray();
   
                    return;
                }
   
                if (renderer.getMap().getLayers().get(i) instanceof TiledMapTileLayer) {
                    layers.add(i);
                }
            }
        }
   
        @Override
        public void draw(SpriteBatch batch, float delta) {
            batch.end();
   
            renderer.render(layerIndeces);
   
            batch.begin();
        }
    }
   

2. Render the Entities. This should be simple enough to do depending on how you have your ECS set up.

3. Render the Upper Map Layers normally, and then to the Stencil Buffer. This is almost identical to the LowerTiledMapRenderSystem, but specifying all layers above the Entities layer instead of below, and also constructing a new ShaderProgram which uses the alphatest shader.

    public class UpperTiledMapRenderSystem extends EntitySystem implements Drawable {
   
        ShaderProgram alpha;
   
        private static final OrthogonalTiledMapRenderer renderer = LevelManager.RENDERER;
   
        private int[] layerIndeces;
   
        public UpperTiledMapRenderSystem() {
            alpha = Assets.getShader("passthrough", "alphatest");
   
            List<Integer> layers = new LinkedList<>();
   
            for (int i = renderer.getMap().getLayers().getCount() - 1; i > 0; i--) {
                if (renderer.getMap().getLayers().get(i).getName().equalsIgnoreCase(Layer.ENTITIES.getName())) {
                    Collections.reverse(layers);
                    layerIndeces = layers.stream().mapToInt(Integer::intValue).toArray();
   
                    return;
                }
   
                if (renderer.getMap().getLayers().get(i) instanceof TiledMapTileLayer) {
                    layers.add(i);
                }
            }
        }
   
        @Override
        public void draw(SpriteBatch batch, float delta) {
            batch.end();
   
            // Render normally
            renderer.render(layerIndeces);
   
            // Enable the Stencil Buffer
            Gdx.gl20.glClear(GL20.GL_STENCIL_BUFFER_BIT);
   
            Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
            Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xFF);
            Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
   
            // Set the Shader of the renderer to the alpha shader
            renderer.getBatch().setShader(alpha);
            // Render the same layers again
            renderer.render(layerIndeces);
            // Disable the shader
            renderer.getBatch().setShader(null);
   
            batch.begin();
        }
    }
   

4. Render the Silhouettes. This is virtually identical to the SpriteRenderSystem, but we're enabling the silhouette shader.

    public class SilhouetteRenderSystem extends SortedIteratingSystem implements Drawable {
   
        private final ShaderProgram silhouetteShader;
   
        public SilhouetteRenderSystem() {
            super(Family.all(
                    // Specify components needed here
            ).get(), new EntityOrderComparator());
   
            silhouetteShader = Assets.getShader("passthrough", "silhouette");
        }
   
        @Override
        protected void processEntity(Entity entity, float deltaTime) {}
   
        @Override
        public void draw(SpriteBatch spriteBatch, float delta) {
            Gdx.gl20.glStencilFunc(GL20.GL_LEQUAL, 0x1, 0xFF);
            Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);
   
            spriteBatch.setShader(silhouetteShader);
   
            // Render your entities here
   
            spriteBatch.setShader(null);
   
            Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);
        }
    }
   

I strongly advise looking into OpenGL Documentation to gain a better understanding of how shaders work, specifically glStencilFunc and glStencilOp.