3
\$\begingroup\$

I've been trying to implement a GPU-based matrix palette skinning algorithm with WebGL, but the rendering appears incorrect even though I can't find evident conceptual problems in the underlying algorithm. Left image is the rigged model in Blender, right image is the same mesh rendered with my algorithm.

blender rig enter image description here

I wrote a custom JSON exporter which exports the exact same data as the THREE.js exporter - position and rotation quaternions for bindposes and keyframes, weights and indices (checked manually). With THREE.js the animation renders correctly, so the issue must lie somewhere in my matrix manipulation process.

My process is the following:

  1. Calculate the world matrices for each bone in the bindpose. This is obtained by first calculating localMatrix with mat4.fromRotationTranslation(localMatrix, bone.rot, bone.pos), then by copying localMatrix in the worldMatrix if the current bone is the root, or by multiplying localMatrix with the parent's worldMatrix if it is not; finally, the inverse bindpose matrix is stored. (* note that, per exporter invariant, each parent precedes all of its children in the array, so each parent's worldMatrix will have been calculated by the time it is requested; see this article).

      for(var i = 0; i < this.geometry.bones.length; i++) {
        var bone = this.geometry.bones[i], localMatrix = mat4.create();
    
        mat4.fromRotationTranslation(localMatrix, bone.rot, bone.pos);
    
        bone.worldMatrix = mat4.create();
        bone.inverseBindpose = mat4.create();
    
        if(bone.parent == -1) {
          mat4.copy(bone.worldMatrix, localMatrix);
        } else {
          // *
          mat4.multiply(bone.worldMatrix, this.geometry.bones[bone.parent].worldMatrix, localMatrix);
        }
        mat4.invert(bone.inverseBindpose, bone.worldMatrix);
      }
    
  2. For each keyframe, recalculate the bone hierarchy with the same algorithm but with the degrees of freedom specified by the keyframe, then for each keyframe-bone calculate a matrix offsetting from the bindpose by multiplying its world matrix with the inverseBindpose calculated in the first step.

      var kf = this.geometry.keyframes;
      for(var i = 0; i < kf.length; i++) {
    
        var flat = [];
    
        for(var j = 0; j < kf[i].length; j++) {
          var bone = kf[i][j],
              parent = this.geometry.bones[j].parent,
              localMatrix = mat4.create();
    
          mat4.fromRotationTranslation(localMatrix, bone.rot, bone.pos);
    
          bone.worldMatrix = mat4.create();
    
          if(parent == -1) {
            mat4.copy(bone.worldMatrix, localMatrix);
          } else {
            mat4.multiply(bone.worldMatrix, kf[i][parent].worldMatrix, localMatrix);
          }
    
          var offsetMatrix = mat4.create();
          mat4.multiply(offsetMatrix, bone.worldMatrix, this.geometry.bones[j].inverseBindpose);
          bone.offsetMatrix = offsetMatrix;
    
          flat.push.apply(flat, offsetMatrix);
        }
        this.keyframes[i] = new Float32Array(flat);
    
      }
    
  3. Plug everything in the buffers and invoke the vertex shader. I tried deforming a mesh in the same way on the CPU and the results are exactly the same, so I think this rules it out and the problem lies in the matrices, but here's the shader for the sake of completeness.

      uniform mat4 uP, uV, uM;
      uniform mat4 uBonesFrame[8];
      uniform mat3 uN;
      uniform bool uSkin;
    
      attribute vec3 aVertex, aNormal;
      attribute vec2 aTexCoord;
    
      attribute highp vec2 aSWeights;
      attribute highp vec2 aSIndices;
    
      varying vec3 vVertex, vNormal;
    
      mat4 boneTransform() {
    
        mat4 ret;
        float normfac = 1.0 / (aSWeights.x + aSWeights.y);
    
        ret = normfac * aSWeights.y * uBonesFrame[int(aSIndices.y)] +
              normfac * aSWeights.x * uBonesFrame[int(aSIndices.x)];
    
        return ret;
      }
    
      void main() {
    
        mat4 bt = uSkin ?
          boneTransform()
          :
          mat4(
              1., 0., 0., 0.,
              0., 1., 0., 0.,
              0., 0., 1., 0.,
              0., 0., 0., 1.
          );
    
        gl_Position = uP * uV * uM * bt * vec4(aVertex, 1.0);
        vVertex = (bt * vec4(aVertex, 1.0)).xyz;
        vNormal = (bt * vec4(aNormal, 0.0)).xyz;
      }
    

In pseudo code, it goes like this:

For each bone
  Calculate localMatrix from quaternion and translation
  If (is root bone) worldMatrix = localMatrix
  Else worldMatrix = parent's worldMatrix * localMatrix
  inverseBindpose = invert(worldMatrix)

For each keyframe
  For each bone
    Calculate localMatrixKF from quaternion and translation
    If (is root bone) worldMatrixKF = localMatrixKF
    Else worldMatrixKF = parent's worldMatrixKF * localMatrixKF
    offsetMatrixKF = worldMatrixKF * inverseBindpose
    Use offsetMatrixKF to deform the vertex

I've also tried deforming (0.0, 0.0, 0.0) with the bones' matrices to obtain the joint positions and I found (0.0, 0.0, 0.0), (-.87, .50, 0.00), (-.87, 2.50, 0.0) and (1.73, 4.00, 0.00) with the offset matrices and (0.0, 0.0, 0.0), (-.87, 1.00, 0.00), (-.87, 1.50, 0.0) and (1.73, 1.00, 0.00) with the world matrices at keyframe 2, where it bends past the XZ plane. The world coordinates look ok to me, I'm not really sure about the offset matrices, the 2.50 and 4.00 values look a little large to me.

Am I doing something incredibly wrong?

If necessary, I'll upload the full code somewhere.

Thank you in advance for your patience, I'm losing my mind over this.


\$\endgroup\$
  • 1
    \$\begingroup\$ Have you tried to refine the problem? As well formatted as the question is, that's an awful lot of code to go through. If you know what the data should be at each step of the process, I'd go through and check all the intermediary values until you find a discrepancy. \$\endgroup\$ – Polar May 3 '14 at 10:30
  • \$\begingroup\$ I'm not really sure what those values should be, actually, I have just a rough, conceptual idea for that but I'll most likely be wrong. I went through most of them, the only odd thing I could notice was the one with the offset matrices I wrote about at the end. But then if I check all the matrices (offset and world) on the first frame, which corresponds to the bindpose, I get all identities which is correct. I'll try ruling out as many things as I can but I've already spent a lot of time trying to debug this, I'm not sure if I can narrow the problem much more than this. \$\endgroup\$ – veeenu May 3 '14 at 13:01
  • \$\begingroup\$ Honestly it looks like your bones are rotating too far. I think that's the reason for distortion. I suggest you check the bone rotation calculations. Further, you can get the coordinates of your transformed points. When the mesh is deformed from the bones in Blender, apply the deformation and then go to edit mode, then you can select a vertex and get its position. Then check the position of the same vertex on your end of things to ensure they match. \$\endgroup\$ – MichaelHouse May 3 '14 at 15:20
  • \$\begingroup\$ I'll look into it asap, didn't think about this approach. Anyway I'm not sure if it's a matter of too much or too little rotation, I think there's some rotten math behind this. I have another model, with similar geometry, rigging and animation which in turn doesn't bend enough to reach the ground. \$\endgroup\$ – veeenu May 3 '14 at 15:32
1
\$\begingroup\$

I solved my own problem. I drew the wireframe of the bones and then I understood. I didn't apply the transformations in Blender so I was deforming a mesh bigger twice than it should have been. See the picture:

example

The armature is clearly smaller than the mesh, this is the reason why the deformation came out incorrect. By applying rotation, location and scale in Blender (Ctrl+A on the 3D view) the problem went away. The algorithm as I described in the question turns out to be fundamentally correct. I hope that it might be of use to someone at least! :)

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.