Here's how the algorithm works:
We want to calculate the "occlusion" at each visible point in the scene, which represents the portion of the hemisphere aligned with the normal at that point that doesn't have geometry to close in that direction, as this is an approximation of the amount of light we would get from the "sky", or more generally the rest of the scene. The real answer is to take an integral over this hemisphere, but this is way too slow for realtime.
Instead, we accumulate the "occlusion" generated by all of the nearby polygons. First, we initialize a buffer to 1, representing no occlusion. We subtract off of this every time we find an occluder. To find what can occlude each pixel, we take every polygon in the scene, extrude it into a volume on one side, and then send that to the rasterizer. What this does is generate a fragment for every scene point (pixel in our buffer) which could possibly be affected by the polygon we are processing. So now our fragment shader knows the position in the scene and normal (from reading the depth buffer/normal buffer) for the point we are calculating AO for, and the details of the polygon (triangle, if you like). So the fragment shader knows all three points of the triangle, not just it's own position.
Now it is just a simple matter to calculate the weighted area of the hemisphere around the pixel that the triangle blocks. We weight it because light coming in from near the normal illuminates the point more than light coming it at an extreme angle, due to the cosine term. We then take this off of the total occlusion using subtractive blending.
In your example, first we would draw the three planes that made up the cube corner, to get a depth/normal buffer. Then we extrude each of these polygons inward. What happens is the points that are far from the other two planes only get touched by their own plane, which gets rejected. The points along the edges have two polygons affect them. Again their own polygon doesn't have any affect, but this time some of the other plane intersects their hemisphere. The closer it is to the edge, the more of the hemisphere is blocked, and the darker the occlusion is. In the corner, each pixel has two non-degenerate polygons, and so they get even darker as more occlusion is subtracted.
Of course, one limitation is that two polygons which both block the same part of the hemisphere will both be counted separately, as the only thing that is tracked between polygons is the total occlusion removed. Without storing something like a mini-cubemap, there would not really be any way to prevent this -- this is a fundamental limitation of this technique. In particular, if you just rendered the same model twice during the "extrude and accumulate" phase, models that were nearby would get twice the occlusion, and be two times darker! Of course, a more realistic scenario is small, thin, repeated geometry, which is probably rare enough to make this a good technique.