A few different things. First, we need to recognize that this is an under-constrained problem. That is, there are many different combinations of thrusters which can fire to result in rotation in the same direction. I'm assuming that in your situation there are only two states for the thrusters, "on", and "off", and all thrusters output equal force.
Second, eyeballing your model, it looks like your "center of mass" is not actually your center of mass. Luckily, this won't effect your calculations for the torque. However, it will effect your calculations for the center of mass displacement. I'm not sure if you care about accuracy on that level though, since your "center of mass" is at least the closest square to the true center of mass.
Third, if you want to calculate how a certain thruster will affect the rotation, you have it right, although you are using an inefficient formula. Torque can be calculated as
r x F, which has magnitude
r*F*sin(theta). However, calculating the angles in this case is an inefficient method. Instead, you should use the cross-product definition of torque directly, as this will be much simpler using the representations you have. Because all your vectors have no z component, the formula for cross product simplifies greatly.
Without changing the results of your computation at all, we can just update your code
private function thrustTorque():Float
var torque = distToCOM.x*dir.y-distToCOM.y*dir.x;
That's much nicer (and faster).
You suggest in your own answer that your solution is to fire all the thrusters with torque in the right direction. Now, that pretty much solves the question that you asked. However, I expect that some point along the line, you're going to find that your strategy is not so satisfactory, if a user holds down the "rotate" button, and all thrusters with a positive torque rotate, potentially moving them on top of rotating them (I'm not sure of the level of detail of your simulation, if you actually do calculate the forces from the thrusters, or if you just visually show them firing and then rotate your model with a constant acceleration or something. Either way, you want the thrusters to be firing at least approximately accurately).
You make no consideration of the net force on the ship. If you had arbitrary thruster amounts, then this could turn into a pretty complicated problem. However, since our thrusters have only two states, it's pretty simple to analyze. I'm not sure exactly what our goal is here, so I could imagine two different ones: first, we want to minimize the total force, while still keeping the torque in the direction we want. Second, we want to maximize the ratio of torque to total force.
As an aside, if you could imagine an additional "thruster volume" control that affects the power of all thrusters simultaneously, then you could set this control so that your two solutions have equal torque, and you see that the second solution can only have a smaller displacement than the first. However, we need to remember that if it is possible to fire the thrusters so that you only rotate and do not move at all, then both solutions will be the same.
So, we're going to go with the second solution, based on the arguments of the previous paragraph. Now, when analyzing the total force, we can simply note that there are only four directions the engines can point. Thus the total force in the x direction is just the number of thrusters pointing left minus the number pointing right, and likewise for the y direction.
After writing up this far, I have to think some more about the algorithm to optimize it. I think the rest of my post is helpful as is, so I'm posting it, but I'll update when I figure out the best way to optimize this configuration (I've thought of a few ways to get approximate answers, but none of them are exact).