Check each individual plane of the AABB against the other AABB. If a plane intersects the other object's AABB, but its opposing plane does not, you know that that plane is involved in the collision. If both opposing faces intersect the other AABB, then the object is fully within the other along that axis, and the direction of collision is ambiguous/undefined.
Practically speaking, these are the same checks used to determine if AABBs collide, so you should be able to determine which faces collide as you do the AABB collision check with little additional cost.
Looking at the example, we see that:
- The sphere's yz+ plane intersects the prism, but yz- does not. Sphere yz+ collides. In a similar manner, the prism's yz- collides.
- Both the prism xy- and xy+ are within the sphere's xy- and xy+. The prism is fully within the sphere in the z direction; collision is ambiguous.
- Both the prism xz- and xz+ are within the sphere's xz- and xy+. The prism is fully within the sphere in the y direction; collision is ambiguous.
So, in this example, we detect a collision between the sphere yz+ and the prism yz-.
The normals of all faces should be obvious (yz+ has (1,0,0), yz- has (-1,0,0), xz- has (0,-1,0) ...) In general, if a quad has the vertices (a, b, c, d) in counter-clockwise order, then your can compute the normal with (b-a).cross(c-b)
Note that this problem does present the possibility of multiple collisions: depending on positioning, you can have collisions in all three axes. I think this 'ambiguity' is pretty inherent, and how you resolve it is really up to you. You might just take the average of the individual normals. You could find the area of intersection in each plane, and then choose the normal which corresponds to the largest area, or use the areas to weight the average of normals. You could find the intrusion depth along each axis, and pick the normal corresponding to the shallowest (or deepest) intrusion. Etc, etc.