Let's consider three gears: \$G_1\$, \$G_2\$ and \$G_3\$.
Now each gear has a given radius \$r_1\$, \$r_2\$ and \$r_3\$.
The problem is to find the angular velocity for each gear (denoted \$\omega_1\$, \$\omega_2\$ and \$\omega_3\$). From a physics course we know that the tangent velocity v is proportional to the radius: \$v_1 = \omega_1 r_1\$.
Now, if a given gear, say \$G_1\$, is connected to a second gear \$G_2\$, then the tangent velocities match: \$v_1=v_2=v\$, where \$v\$ is the overall velocity of the system, a global parameter for a system of connected gears.
Now it follows that \$\omega_1r_1 = v\$ so \$\omega_1 = \frac{v}{r_1}\$ for each gear. That way you can find out the angular velocity of all gears in a system.
The next step is finding the rotation direction. For this, simply choose a gear (say G_1), fix the rotation direction (say to \$+\omega_1\$), look which gears are connected to it. If \$G_2\$ is connected to it, its angular velocity is \$-\omega_2\$.
There is still the problem of the clinch situation:
For small-scale problems I would advise to simply do this gear-wise for all gears: Then, if for any gear you want to set a different direction than already set - set all angular velocity values (or the system velocity) to 0 - the system is "jammed"!
You simply do this for each set of connected gears.
Its not a very sophisticated method to do this but I think it should work. :-)
One more comment: you can easily save the gear structure in the so-called "adjacency matrix". It is a matrix that is 1, if gear i is connected to gear j and 0 otherwise. As an example, take the 3x3 matrix consisting of only ones - where each of three gears is connected to the other - it represents a jammed system!
In your scripts, use the adjacency matrix to write the script that determines the rotation directions!