GLSL - Only draw arond point / flashlight effect

Question

I'm currently trying to achieve a sort of flashlight effect in GLSL. I only want to draw textures within a certain distance of a specified point (the mouse position, for example). Here's a quick example drawing of the effect I wish to achieve:

I've successfully created a shader that checks the position from a certain point to the position of the fragment, and then discards the fragment if the distance is larger than the specified light radius. The frag shader looks like this:

#version 400

#ifdef GL_ES
precision mediump float;
#endif

in vec4 v_color;
in vec2 v_texCoords;
in vec2 v_position;

out vec4 outColor;

uniform sampler2D u_texture;
uniform vec2 u_mousePos;
uniform float u_lightRadius;

void main() {
    float distance = distance((v_position + 1.0) / 2.0, u_mousePosition);
    if(distance > u_lightRadius) {
        discard;
    }

    outColor = v_color * texture2D(u_texture, v_texCoords);
}

The problem with this shader is that the visible area isn't a circle, it's an elipse. This is obviously because the screen aspect ratio is not 1:1. I've tried multiplying the Y values of all of the position variables with a resolution uniform, but it still remains an elipse.

My other problem is that I wish to have smooth edges on my visible area. I've read up some on the subject and I see that you can use mix and/or smoothstep to achieve this, but I'm unsure on how to implement it.

Anyone got any suggestions?

Answer 1

To fix the aspect ratio (and speed up the shader a bit):

void main() {
    vec2 diff = (v_position + 1.0) * 0.5 - u_mousePosition;

    diff.x *= u_aspect;  // e.g.: (16.0 / 9.0) for 16:9

    float distance = dot(diff, diff); // square of distance, saves a square-root

    if(distance > u_lightRadius_squared)) {
        discard;
    }

    outColor = v_color * texture2D(u_texture, v_texCoords);
}

To get a smooth edge remove the if(...)discard and add at the end:

outColor.a *= 1.0 - smoothstep(u_lightRadius_minus_one_pixel, u_lightRadius_plus_one_pixel, sqrt(distance));

For a less perfect but faster version:

outColor.a *= 1.0 - smoothstep(u_lightRadius_minus_one_pixel_squared, u_lightRadius_plus_one_pixel_squared, distance);

You can add back if(outColor.a <= 0) discard; at the very end to eliminate transparent pixels.