4
\$\begingroup\$

I'm trying to render a silhouette of any entity behind foreground objects to indicate their position to the player. Here's an example of what I want to achieve, from Hyper Light Drifter:

Hyper Light Drifter Silhouette

From what I've read online about GLSL and the ways I might be able to achieve this, the closest thing I can think of to achieve this would be to use the stencil buffer, which was also indicated in this question:

2D Hidden Object Silhouette Shader

I don't quite understand how I can achieve this within LibGDX. Should I be using a vertex / fragment shader, or the native Gdx methods? This is my current rendering workflow:

Render Lower Tiled Map Layers
Render Entities
Render Upper Tiled Map Layers

From what I understand, I would have to:

  1. Write the top layer to the stencil buffer
  2. Render the entities (silhouettes) within the stencil

Is this right? If anyone can enlighten me, it would be greatly appreciated!

\$\endgroup\$
2
\$\begingroup\$

All of the drawing code is separated into their own systems responsible for rendering what is on the screen. Differentiate between Lower and Upper layers by naming the layer you want the Entities to be on. I use a custom interface designed to separate the render method of LibGDX into update(delta) and draw(SpriteBatch, delta), similar to C#'s XNA as this was more familiar for me.

You need two fragment shaders and one vertex shader to get the desired result. The vertex shader is a simple passthrough shader:

    attribute vec4 a_color;
    attribute vec4 a_position;
    attribute vec2 a_texCoord0;
    
    uniform mat4 u_projTrans;
    
    varying vec4 v_color;
    varying vec2 v_texCoord0;
    
    void main() {
        v_color = a_color;
        v_color.a = v_color.a * (255.0/254.0);
        v_texCoord0 = a_texCoord0;
    
        gl_Position = u_projTrans * a_position;
    }

Then you need a fragment shader to discard any pixels from the upper layers with an alpha value of 0 from the Stencil Buffer:

    varying vec4 v_color;
    varying vec2 v_texCoord0;
    
    uniform sampler2D u_texture;
    
    void main() {
        gl_FragColor = v_color * texture2D(u_texture, v_texCoord0);
    
        if (gl_FragColor.a == 0.0) {
            discard;
        }
    }

And finally, another fragment shader to render entity silhouettes:

    varying vec4 v_color;
    varying vec2 v_texCoord0;
    
    uniform sampler2D u_texture;
    
    void main() {
        gl_FragColor = vec4(0.0, 0.75, 1.0, 0.25) * texture2D(u_texture, v_texCoord0).a;
    }
  1. Render Lower Tiled Map Layers.

     public class LowerTiledMapRenderSystem extends EntitySystem implements Drawable {
    
         private int[] layerIndeces;
    
         private static final OrthogonalTiledMapRenderer renderer = LevelManager.RENDERER;
    
         public LowerTiledMapRenderSystem() {
             List<Integer> layers = new LinkedList<>();
    
             for (int i = 0; i < renderer.getMap().getLayers().getCount(); i++) {
                 if (renderer.getMap().getLayers().get(i).getName().equalsIgnoreCase("entities")) {
                     layerIndeces = layers.stream().mapToInt(Integer::intValue).toArray();
    
                     return;
                 }
    
                 if (renderer.getMap().getLayers().get(i) instanceof TiledMapTileLayer) {
                     layers.add(i);
                 }
             }
         }
    
         @Override
         public void draw(SpriteBatch batch, float delta) {
             batch.end();
    
             renderer.render(layerIndeces);
    
             batch.begin();
         }
     }
    
  2. Render the Entities. This should be simple enough to do depending on how you have your ECS set up.

  3. Render the Upper Map Layers normally, and then to the Stencil Buffer. This is almost identical to the LowerTiledMapRenderSystem, but specifying all layers above the Entities layer instead of below, and also constructing a new ShaderProgram which uses the alphatest shader.

     public class UpperTiledMapRenderSystem extends EntitySystem implements Drawable {
    
         ShaderProgram alpha;
    
         private static final OrthogonalTiledMapRenderer renderer = LevelManager.RENDERER;
    
         private int[] layerIndeces;
    
         public UpperTiledMapRenderSystem() {
             alpha = Assets.getShader("passthrough", "alphatest");
    
             List<Integer> layers = new LinkedList<>();
    
             for (int i = renderer.getMap().getLayers().getCount() - 1; i > 0; i--) {
                 if (renderer.getMap().getLayers().get(i).getName().equalsIgnoreCase(Layer.ENTITIES.getName())) {
                     Collections.reverse(layers);
                     layerIndeces = layers.stream().mapToInt(Integer::intValue).toArray();
    
                     return;
                 }
    
                 if (renderer.getMap().getLayers().get(i) instanceof TiledMapTileLayer) {
                     layers.add(i);
                 }
             }
         }
    
         @Override
         public void draw(SpriteBatch batch, float delta) {
             batch.end();
    
             // Render normally
             renderer.render(layerIndeces);
    
             // Enable the Stencil Buffer
             Gdx.gl20.glClear(GL20.GL_STENCIL_BUFFER_BIT);
    
             Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
             Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xFF);
             Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    
             // Set the Shader of the renderer to the alpha shader
             renderer.getBatch().setShader(alpha);
             // Render the same layers again
             renderer.render(layerIndeces);
             // Disable the shader
             renderer.getBatch().setShader(null);
    
             batch.begin();
         }
     }
    
  4. Render the Silhouettes. This is virtually identical to the SpriteRenderSystem, but we're enabling the silhouette shader.

     public class SilhouetteRenderSystem extends SortedIteratingSystem implements Drawable {
    
         private final ShaderProgram silhouetteShader;
    
         public SilhouetteRenderSystem() {
             super(Family.all(
                     // Specify components needed here
             ).get(), new EntityOrderComparator());
    
             silhouetteShader = Assets.getShader("passthrough", "silhouette");
         }
    
         @Override
         protected void processEntity(Entity entity, float deltaTime) {}
    
         @Override
         public void draw(SpriteBatch spriteBatch, float delta) {
             Gdx.gl20.glStencilFunc(GL20.GL_LEQUAL, 0x1, 0xFF);
             Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);
    
             spriteBatch.setShader(silhouetteShader);
    
             // Render your entities here
    
             spriteBatch.setShader(null);
    
             Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);
         }
     }
    

I strongly advise looking into OpenGL Documentation to gain a better understanding of how shaders work, specifically glStencilFunc and glStencilOp.

\$\endgroup\$
3
  • \$\begingroup\$ Hello could you please share a class? Because the code above doesn't work for me. I'll be very grateful! \$\endgroup\$ – Ra Aga Sep 12 '20 at 0:13
  • \$\begingroup\$ @RaAga note that this answer was posted 5 years ago - it's quite possible the user might not see this comment, or no longer has the code for this solution. So you may be better off posting a new question of your own, using the "Ask Question" button. You can include a link to this post for context if it helps. It's also always a good idea to describe the specific problem you're having, not just "the code doesn't work for me". The more precisely you can describe what you tried and what went wrong, the better we can target useful solutions. \$\endgroup\$ – DMGregory Sep 12 '20 at 1:48
  • \$\begingroup\$ @RaAga I have provided a more detailed answer to how the result can be achieved, as I admit it was a slightly obscure answer. I have provided links to the OpenGL documentation and methods used to encourage others to read. \$\endgroup\$ – driima Oct 6 '20 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.