I need insect-like agents that have fallen/rolled on their back/side to naturally re-orient themselves. The agents are modeled as simple cuboids in the physics engine (Bullet) to approximate an ant-like shape.
At the moment I correct the orientation by instantly altering it when the dot product of surface 'up' and agent 'up' is less than some number (e.g. -0.5). Thus the agents can fall over only so much before they are insta-righted in the blink of an eye.
Obviously this looks nasty. Plus in the border cases, where the dot product isn't quite low enough, they can still fall over and don't get self-righted.
So, I need a way to have the critters automatically roll back onto their feet. How can I do this? The way I see it there are two problems:
1) How to determine the best rotation axis (e.g. better to roll on their back than head-to-tail)?
2) How to work out the magnitude of torque required to overcome the physics engine, whilst not being too much that has them flipping end over end?
[Full disclosure: I originally posted this question on gamedev.net, to no response. I also posted in StackOverflow, but deleted the post when this stackexchange was recommended instead]