Implementing skeletal animation and GPU skinning (DOOM MD5)

Question

DISCLAIMER
This question has been completely rewritten to narrow the scope of the question in light of previous suggestions and answers, but the same problem still persists.

Currently I'm trying to implement skeletal animation (GPU skinning) in my project.

So far I have achieved single joint translation and rotation, and multi-jointed translation. The problem arises when I try to rotate a multi-jointed skeleton.

At the moment I'm following quite a few sources in trying to implement skeletal animation. The MD5 spec, a blog post with source code and I've also seen many other tutorials.

Currently when I try and load the popular "Bob Lamp" model, I get this result.

Some information about my implementation:

• My maths is ordered with "vector to the left"
• The vectors and rotations of animation frames are being converted into matrices
• I pre-multiply the inverse bind pose and pose matrices

Here is my shader code.

#version 330 core

smooth out vec2 vVaryingTexCoords;
smooth out vec3 vVaryingNormals;
smooth out vec4 vWeightColor;

uniform mat4 MV;
uniform mat4 MVP;
uniform mat4 Pallete[55];

layout(location = 0) in vec3 vPos;
layout(location = 1) in vec2 vTexCoords;
layout(location = 2) in vec3 vNormals;
layout(location = 3) in int vSkeleton[4];
layout(location = 4) in vec3 vWeight;

void main()
{
    vec4 wpos2 = vec4(0.0);
    vec4 norm2 = vec4(0.0);
    vec4 wpos = vec4(vPos, 1.0);
    vec4 norm = vec4(vNormals, 0.0);
    vec4 weight = vec4(vWeight, (1.0f-(vWeight[0] + vWeight[1] + vWeight[2])));

    mat4 BoneTransform = mat4(0.0);
    for(int i = 0; i < 4; i++) {
        float fW = weight[i];

        if(fW > 0) {
            int jI = vSkeleton[i];
            vec4 tmpPos = wpos * (Pallete[jI]);
            wpos2 += fW * tmpPos;
        }
    }

    wpos2 /= wpos2.w;

    vWeightColor = weight;
    vVaryingTexCoords = vTexCoords;
    vVaryingNormals = normalize(vec3(vec4(vNormals, 0.0) * MV));
    gl_Position = wpos2 * MVP;
}

Another peculiar thing to mention is that multi-jointed animation works when the vertices are inline with the bones. As soon as the vertices move away from the bones, they deform incorrectly.

UPDATE
In response to @MickLH

My shader code now has wpos2 /= wpos2.w after the for loop. This is the result.

Answer 1

Your layout is illegal, vSkeleton[4] needs 4 indices since it is a 4-part array.

layout(location = 3) in int vSkeleton[4];
layout(location = 4) in vec3 vWeight;

should be:

layout(location = 3) in int vSkeleton[4];
layout(location = 7) in vec3 vWeight;

or better yet you should use glGetUniformLocation and let the GLSL compiler assign them for your:

in int vSkeleton[4];
in vec3 vWeight;

Answer 2

I'm not sure if this will solve your problem, but at least I can point some lines in your code, that I don't belive are correct.

First of all, you need to decide how you're going to multiply your matrices and vectors, i.e. are you going to use the left-side or right-side convention. In your shader code you have this line:

wpos = BoneTransform * wpos;

which suggests "vectors on the right", but then you have this:

gl_Position = wpos * MVP;

so it simply does not match. So, you either need to change the first on or the second one. But does it matter which one? Since you're calling your matrix MVP I will assume that this is equal to

ModelViewMatrix * ProjectionMatrix

and in that case you should probably use the "vectors on the left" convention. It's not very common but that's totally fine to me. So it may look like the only thing you need to fix is wpos = wpos * BoneTransform. Indeed, the formula

BoneTransform += (invBindPose[vSkeleton[i]] * Pallete[vSkeleton[i]]) * weight[i];

is good enough for left-side vector multiplication, at least if the Pallete matrices are computed the right way, but are they?

In fact, this is determined by the MD5mech file format itself. I've look throught their documentation but I'm not entirely sure which convention they're actually using. BTW, because of this ambiguity, it's always better to use 4x4 matrices than quaternion + vector pair. In your situation this is a matter of checking wheter you should use pqi instead of pq in the formula

Math3::quat::RotateVector3(rpos, pq, jv);

or not. Also another line

Math3::quat nq = pq * jq;

could be nq = jq * pq as well, so in fact you have at most 4 cases to check. Probably the easiest way to go is to expermient, but I would encourage you to look in the source code of the Math3 library to determine if their RotateVector3 routine behaves as described in the MD5mesh docs.

Also have in mind that there are at least two conventions to store quaternions in memory, which differ in the order of quaternion's coordinates. The natural one is w,x,y,z but I've observed that some engines/libraries tends to place the w coordinate at the end of the sequence. That is usually fine, unless you're forced to convert from vec4 to quaternion or vice-versa. From my experiance it can be a real pain to look for errors of this type, so be carefull with that.

@akaltar Please note, that *= is used for composing transformations. Here we only want to compute a baricentric combination of some affine transformations, so in the formula for BoneTransform the use of += is totally fine.

Answer 3

1. normalize(weight); is wrong as it does not make the components add up to one, it makes the length of the vector one which is not what you want. But as you do not use it's return value this line doesn't do anything. This line is probably not required from what I understand of your setup.

2. BoneTransform += makes no sense. Adding matrixes together does not concatenate the transforms, you need to multiply them. But this isn't correct for another reason:

You may not apply bone weights to the transformations, you have to apply them to the transformed vertex:

for(int i = 0; i < 4; i++) {
    if(vSkeleton[i] != -1) {
        if(i == 0) {
            BoneTransform = ((invBindPose[vSkeleton[i]] * Pallete[vSkeleton[i]]) * weight[i]);
            // BoneTransform = ((Pallete[vSkeleton[i]] * invBindPose[vSkeleton[i]]) * weight[i]);
        } else {
            BoneTransform += ((invBindPose[vSkeleton[i]] * Pallete[vSkeleton[i]]) * weight[i]);
            // BoneTransform += ((Pallete[vSkeleton[i]] * invBindPose[vSkeleton[i]]) * weight[i]);
        }
    }
}

wpos = BoneTransform * wpos;

Has to look more like this:

wpos = 0.0;
for(int i = 0; i < 4; i++) 
{
    if(vSkeleton[i] != -1) 
    {
        BoneTransform = invBindPose[vSkeleton[i]] * Pallete[vSkeleton[i]];
        wpos += ( BoneTransform * vec4(vPos, 1.0) ) * weight[i];
    }
}

This is one reason why hlsl's mul function is better thant glsl's * operator. You may not switch around the operands when doing (matrix * vector) * scalar.

I'm not sure if I caught all the problems. There could still be problems on the cpu side and how your pass parameters. Let me know how it goes

3. PERFORMANCE: You should remove the binding pose from the bone transformation on the cpu. This has to be done for each vertex on the gpu and only once for each bone when done on the cpu.