Ambient occlusion: Correct culling after anisotropy correction

Question

I've added simple color based ambient occlusion to my game.

First it looked like this (from giawa.com):

The description: "Due to anisotropy, the ambient occlusion shading on the top left is inconsistent with the top right. We need to flip those quads so that they get rendered properly."

So according to that I had to flip some faces in order to fix the issue. I did and then it would look like this:

This is of course because of back face culling, culling the flipped faces.

When I disable culling, it looks like this:

Which is how it's supposed to look. But as you can imagine, that costs a lot of performance because culling is now disabled.

According to the article: http://www.giawa.com/author/giawa/page/10/ "This occurs due to differences in how the GPU linear interpolates across the edge of a triangle versus the body of a triangle. This is called anisotropy, and can be fixed by ‘flipping’ the quad. My final fix was to enable polygon culling, which meant I had to make sure to render my quads in a consistent winding. "

I've searched for this "polygon culling" but could not find anything about that.

I can't just switch between culling modes because all the vertexes are added to a vertex buffer when initializing a chunk. This is so I can render each chunk in one draw call.

I hope anyone knows a way to fix this, thanks in advance.

Answer 1

By "flip the quads", the article doesn't mean to reverse the winding order. That swaps the front and back faces of the quad and causes other problems, as you've found.

The article means to change the way the quad is divided into two triangles. Instead of a quad like this:

 __________
|         *|
|      *   |
|   *      |
|*_________|

sometimes you need a quad like this:

 __________
|*         |
|   *      |
|      *   |
|_________*|

Splitting along the other diagonal makes the vertex interpolation look different, and the article is taking advantage of this to improve the appearance of the AO.

It can be done by permuting the vertices. If the first quad had vertices 1, 2, 3, 4 then the second one should have 2, 3, 4, 1.

(BTW, the author's use of the term "anisotropy" in that article is a bit confusing as that word is usually used only to talk about anisotropic texture filtering, which is something completely different. What the author calls "anisotropy" I would just call the fact that vertex interpolation is done per-triangle, not per-quad.)