To put it simply - winding order. Winding order is the way you define which side of a triangle is the front and which side is the back. For example, OpenGL's default winding order is counter-clockwise, which means the triangle on the left is pointed out of the screen and the triangle on the right is pointed into the screen. If you had back-face culling turned on you would not be able to see the right-hand triangle.
| \ | \
| \ | \
This works together with the non-commutative property of the vector cross product
a × b = -(b × a)
to ensure that when you calculate the normal of a triangle it points in the right direction. To use the triangle example from above, if you always calculate your normal vector as follows, the normal will always point out from the front of the triangle (towards you for the left triangle, away for the right). Note that to obtain a unit normal, which is desirable for lighting and collision, you also need to normalize the vector.
n = (p2 - p1) × (p3 - p1)
Since the vector cross product is conventionally right-handed, you can verify the normal direction yourself using this method with your right hand:
- Point your index finger in the direction of the left-hand operand
- Point your middle finger in the direction of the right-hand operand
- Your thumb is now pointing in the direction of the vector obtained with the cross product
a × b
Notice how if you switch the operands your thumb ends up in the opposite direction.
So how does this relate to collision detection?
To ensure that you always obtain a collision normal that will point away from the surface, only detect collisions from the front side. If, like in your example, you want to be able to collide with both sides of an object, put two collision surfaces, one flipped, on top of each other. Again, refer back to the two triangles and how the winding order defines which side is the front. If you layer the two triangles, now you have a double-sided triangle with a different normal (the exact opposite!) on each side. Since you only detect collisions with the "front" of a surface you always obtain the correct normal.
Respecting winding order is how convex hull algorithms are still able to determine which is the "outside" despite not being given that information explicitly. The Quickhull algorithm uses tetrahedra that are eventually used to build a triangle mesh, and if the winding order is maintained properly from the start all the normals will be facing in the correct direction.