# GPU skinning algorithm incorrectly distorts mesh

I've been trying to implement a GPU-based matrix palette skinning algorithm with WebGL, but the rendering appears incorrect even though I can't find evident conceptual problems in the underlying algorithm. Left image is the rigged model in Blender, right image is the same mesh rendered with my algorithm.

I wrote a custom JSON exporter which exports the exact same data as the THREE.js exporter - position and rotation quaternions for bindposes and keyframes, weights and indices (checked manually). With THREE.js the animation renders correctly, so the issue must lie somewhere in my matrix manipulation process.

My process is the following:

1. Calculate the world matrices for each bone in the bindpose. This is obtained by first calculating localMatrix with mat4.fromRotationTranslation(localMatrix, bone.rot, bone.pos), then by copying localMatrix in the worldMatrix if the current bone is the root, or by multiplying localMatrix with the parent's worldMatrix if it is not; finally, the inverse bindpose matrix is stored. (* note that, per exporter invariant, each parent precedes all of its children in the array, so each parent's worldMatrix will have been calculated by the time it is requested; see this article).

  for(var i = 0; i < this.geometry.bones.length; i++) {
var bone = this.geometry.bones[i], localMatrix = mat4.create();

mat4.fromRotationTranslation(localMatrix, bone.rot, bone.pos);

bone.worldMatrix = mat4.create();
bone.inverseBindpose = mat4.create();

if(bone.parent == -1) {
mat4.copy(bone.worldMatrix, localMatrix);
} else {
// *
mat4.multiply(bone.worldMatrix, this.geometry.bones[bone.parent].worldMatrix, localMatrix);
}
mat4.invert(bone.inverseBindpose, bone.worldMatrix);
}

2. For each keyframe, recalculate the bone hierarchy with the same algorithm but with the degrees of freedom specified by the keyframe, then for each keyframe-bone calculate a matrix offsetting from the bindpose by multiplying its world matrix with the inverseBindpose calculated in the first step.

  var kf = this.geometry.keyframes;
for(var i = 0; i < kf.length; i++) {

var flat = [];

for(var j = 0; j < kf[i].length; j++) {
var bone = kf[i][j],
parent = this.geometry.bones[j].parent,
localMatrix = mat4.create();

mat4.fromRotationTranslation(localMatrix, bone.rot, bone.pos);

bone.worldMatrix = mat4.create();

if(parent == -1) {
mat4.copy(bone.worldMatrix, localMatrix);
} else {
mat4.multiply(bone.worldMatrix, kf[i][parent].worldMatrix, localMatrix);
}

var offsetMatrix = mat4.create();
mat4.multiply(offsetMatrix, bone.worldMatrix, this.geometry.bones[j].inverseBindpose);
bone.offsetMatrix = offsetMatrix;

flat.push.apply(flat, offsetMatrix);
}
this.keyframes[i] = new Float32Array(flat);

}

3. Plug everything in the buffers and invoke the vertex shader. I tried deforming a mesh in the same way on the CPU and the results are exactly the same, so I think this rules it out and the problem lies in the matrices, but here's the shader for the sake of completeness.

  uniform mat4 uP, uV, uM;
uniform mat4 uBonesFrame[8];
uniform mat3 uN;
uniform bool uSkin;

attribute vec3 aVertex, aNormal;
attribute vec2 aTexCoord;

attribute highp vec2 aSWeights;
attribute highp vec2 aSIndices;

varying vec3 vVertex, vNormal;

mat4 boneTransform() {

mat4 ret;
float normfac = 1.0 / (aSWeights.x + aSWeights.y);

ret = normfac * aSWeights.y * uBonesFrame[int(aSIndices.y)] +
normfac * aSWeights.x * uBonesFrame[int(aSIndices.x)];

return ret;
}

void main() {

mat4 bt = uSkin ?
boneTransform()
:
mat4(
1., 0., 0., 0.,
0., 1., 0., 0.,
0., 0., 1., 0.,
0., 0., 0., 1.
);

gl_Position = uP * uV * uM * bt * vec4(aVertex, 1.0);
vVertex = (bt * vec4(aVertex, 1.0)).xyz;
vNormal = (bt * vec4(aNormal, 0.0)).xyz;
}


In pseudo code, it goes like this:

For each bone
Calculate localMatrix from quaternion and translation
If (is root bone) worldMatrix = localMatrix
Else worldMatrix = parent's worldMatrix * localMatrix
inverseBindpose = invert(worldMatrix)

For each keyframe
For each bone
Calculate localMatrixKF from quaternion and translation
If (is root bone) worldMatrixKF = localMatrixKF
Else worldMatrixKF = parent's worldMatrixKF * localMatrixKF
offsetMatrixKF = worldMatrixKF * inverseBindpose
Use offsetMatrixKF to deform the vertex


I've also tried deforming (0.0, 0.0, 0.0) with the bones' matrices to obtain the joint positions and I found (0.0, 0.0, 0.0), (-.87, .50, 0.00), (-.87, 2.50, 0.0) and (1.73, 4.00, 0.00) with the offset matrices and (0.0, 0.0, 0.0), (-.87, 1.00, 0.00), (-.87, 1.50, 0.0) and (1.73, 1.00, 0.00) with the world matrices at keyframe 2, where it bends past the XZ plane. The world coordinates look ok to me, I'm not really sure about the offset matrices, the 2.50 and 4.00 values look a little large to me.

Am I doing something incredibly wrong?

If necessary, I'll upload the full code somewhere.

Thank you in advance for your patience, I'm losing my mind over this.

• Have you tried to refine the problem? As well formatted as the question is, that's an awful lot of code to go through. If you know what the data should be at each step of the process, I'd go through and check all the intermediary values until you find a discrepancy. – Polar May 3 '14 at 10:30
• I'm not really sure what those values should be, actually, I have just a rough, conceptual idea for that but I'll most likely be wrong. I went through most of them, the only odd thing I could notice was the one with the offset matrices I wrote about at the end. But then if I check all the matrices (offset and world) on the first frame, which corresponds to the bindpose, I get all identities which is correct. I'll try ruling out as many things as I can but I've already spent a lot of time trying to debug this, I'm not sure if I can narrow the problem much more than this. – veeenu May 3 '14 at 13:01
• Honestly it looks like your bones are rotating too far. I think that's the reason for distortion. I suggest you check the bone rotation calculations. Further, you can get the coordinates of your transformed points. When the mesh is deformed from the bones in Blender, apply the deformation and then go to edit mode, then you can select a vertex and get its position. Then check the position of the same vertex on your end of things to ensure they match. – MichaelHouse May 3 '14 at 15:20
• I'll look into it asap, didn't think about this approach. Anyway I'm not sure if it's a matter of too much or too little rotation, I think there's some rotten math behind this. I have another model, with similar geometry, rigging and animation which in turn doesn't bend enough to reach the ground. – veeenu May 3 '14 at 15:32