3D Physics computations are messy and unwieldy. There is no avoiding that.
However, you still have a couple of choices, once you are comfortable bouncing between coordinate systems.
- Compute your impulses locally, which means inertia tensor never needs to be changed. To do this means transforming contact points to local space for each body.
M = inv(M), cpLocal = M * cp, J = compute impulse, W += I * J
- Compute your impulses in world space, then transform then to local space for each body. This means taking the inverse of your model transform and multiplying the impulse by it.
Mi = mat3( inv( M ) ), Jlocal = Mi * J, W += I * Jlocal
- Compute in world space, and transform the inertia tensor to world space then use that to adjust the angular effects. This is :
M3I = M3( rot_toM4 ) * I, W += M3I * J
I will point out, by the way, that many engines do not try to model the inertia of a shape perfectly, and will fudge the numbers, so no rotation of the inertia tensor is necessary.