I'm having a problem with the generation of a shadowmap that is converted to RGBA. The following glsl code is used to put the depth into the depthbuffer :

 * float/rgba8 encoding/decoding so that we can use an RGBA8
 * shadow map instead of floating point render targets which might
 * not be supported everywhere
 * http://aras-p.info/blog/2009/07/30/encoding-floats-to-rgba-the-final/
 * http://www.gamedev.net/topic/486847-encoding-16-and-32-bit-floating-point-value-into-rgba-byte-texture/
 * Only store values between 0.0 and 1.
vec4 encodeDepth(float floatValue) {
    const float toFixed = 255.0 / 256;
    vec4 comp;
    comp.r = fract(floatValue * toFixed * 1);
    comp.g = fract(floatValue * toFixed * 255);
    comp.b = fract(floatValue * toFixed * 255 * 255);
    comp.a = fract(floatValue * toFixed * 255 * 255 * 255);
    return comp;

float decodeDepth(vec4 rgba) {
    const float fromFixed = 256.0 / 255;
    float val = rgba.r * fromFixed / (1);
    val += rgba.g * fromFixed / (255);
    val += rgba.b * fromFixed / (255 * 255);
    val += rgba.a * fromFixed / (255 * 255 * 255);
    return val;

The code that creates the depth buffer is the following:

bool DepthBuffer::init(const glm::ivec2& dimension) {
    _dimension = dimension;

    glGenFramebuffers(1, &_fbo);
    GL_setName(GL_FRAMEBUFFER, _fbo, "depthfbo");
    ScopedFrameBuffer scopedFrameBuffer(_fbo);

    glGenTextures(1, &_depthTexture);
    GL_setName(GL_TEXTURE, _depthTexture, "depthtexture");
    glBindTexture(GL_TEXTURE_2D, _depthTexture);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, 0);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, dimension.x, dimension.y, 0, GL_RGBA, GL_UNSIGNED_BYTE, nullptr);
    glBindTexture(GL_TEXTURE_2D, 0);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, _depthTexture, 0);

    const GLenum drawBuffers[] = { GL_COLOR_ATTACHMENT0 };
    glDrawBuffers(SDL_arraysize(drawBuffers), drawBuffers);

    const GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    if (status != GL_FRAMEBUFFER_COMPLETE) {
        Log::error("FB error, status: %i", (int)status);
        return false;

    return true;

I am calculating the light space positions in the worlds vertex shader like this:

v_lightspacepos = u_light * worldpos_vertex;

u_light is a the projection*view matrix of the sun light camera.

v_lightspacepos is then used to perform the shadow sampler lookup via:

vec2 calculateShadowUV() {
    // convert from -1, 1 to tex coords in the range 0, 1
    vec2 uv = v_lightspacepos.xy / v_lightspacepos.w;
    return uv * 0.5 + 0.5;

float calculateShadow(float ndotl) {
    vec2 smUV = calculateShadowUV();
    float depth = v_lightspacepos.z;
    return sampleShadowPCF(u_shadowmap, smUV, u_screensize, depth, ndotl);

calculateShadow is then used like this:

float ndotl = dot(normal, u_lightdir);
float shadow = calculateShadow(ndotl);

The sun light camera projection matrix is calculated like this:

static const glm::mat4 normalizeDepth = glm::scale(glm::translate(glm::mat4(), glm::backward), glm::vec3(1.0f, 1.0f, 0.5f));
_projectionMatrix = normalizeDepth * glm::ortho(bbox.getMinX(), bbox.getMaxX(), bbox.getMinZ(), bbox.getMaxZ(), nearPlane(), farPlane());

Does anyone see anything wrong here? I'm debugging this code since days now and am not able to identify the problem. See the attached screenshot for the error. The black area will move with the camera. Shadows are (somewhat) working except for the black area and the "wrong" shadowmap.

World render with depth map render


I'm not sure Why are you using RGBA8 for a shadow map... The shadow map algorithm relies on the depth buffer, so you can initialize a texture of type GL_DEPTH_COMPONENT and if you want floating point you can GL_DEPTH_COMPONENT32F, this should be supported on most platforms including OpenGL ES 3 (at least on ARM GPUs)

I would also advise you to use GL_COMPARE_REF_TO_TEXTURE when doing the shadow map so you don't have to do the actual comparison yourself in shader code. The superbible explains shadow mapping and provides code, it can also be found here https://github.com/openglsuperbible/sb7code/blob/master/src/shadowmapping/shadowmapping.cpp

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.