Please think of this in an engine-agnostic way first, and then we can talk about specific engines.
In my game, I have a 3D character which is wearing a hakama (think of it as a long pleated skirt). This character will be doing a lot of dancing, which I'm getting a lot of motion capture data for. This is how my character is rigged right now (view from Unity):
This character has a standard 3D Studio Max humanoid rig, plus a bunch of bones for the "dangling" stuff such as the skirt, the sleeves, the long hair and other parts.
Unfortunately, motion capture can only give me data for the humanoid part, not for the additional parts, so I need a way to animate these. For now, let's focus on the hakama (skirt). I have a few ways of doing this:
Removing the additional bones, and rig the cloth directly to the leg bones: This is my last resort, but one that I'd like to avoid, as the hakama will move directly with the legs as if it were a tight fit, losing a lot of realism.
Keep the bones and animate them manually. There are just too many frames and too many characters for it to feasible to do this manually, so this is out of the question.
Simulate the leg and cloth interaction in some way. Fortunately, the dancing is static (just for cutscenes), so I can do this offline, export the resulting bone transforms for each frame, and apply this to the character during playback.
Since Unity has a pretty mature physics engine, I am trying to hack an offline physics simulation that exports the resulting bone transforms for each frame, which I can then playback in real time in the actual game (which by the way is not Unity-based). So far, the sleeves and hair have shown some pretty good results, but the hakama is much more difficult, and I haven't gotten it to work even close to decently.
I have tried:
Setting colliders in the legs and the skirt, so when the legs push the skirt, the skirt moves. I can set a bunch of joints to set some constraints, but setting a correct shape and position for the skirt colliders is not trivial, since the leg colliders eventually find a hole between the skirt colliders, which leads to the leg intersecting with the skirt.
Instead of colliders, set spring joints between the leg bones and the skirt bones. Unfortunately, springs don't have a concept of "pushing" vs "pulling", and I can visualize a lot of cases where the simulation will lead to the leg intersecting the skirt.
Notice that I can't use Unity's cloth simulator, since this acts on polygons directly, and I need to be able to export the results of the simulation in terms of bone matrices.
It's not mandatory to use Unity. I just need to find a way to simulate the leg-cloth interaction and export the results in a series of bone transforms.
Any ideas on how to solve this problem?