Enhancing ambient occlusion via vert colors on vertex lit scene

We have a deformable terrain surface. Initially ambient occlusion was baked, but since the terrain can deform in real time, this approach isn't practical anymore. Therefore we used vertex paint to imitate occlusion based on the angle of the surrounding terrain vertices. We will then multiply the vertex color with the terrain texture and get a relatively cheap ambient occlusion.

Below are two examples of our terrain. The second has applied occlusion based on surrounding vertices, as mentioned before. But unfortunately it seems like the lower a set of vertices is in relation to the surrounding vertices (or the tighter the crease), the darker the vertex will be.

Vertex lighting in general works, but the results aren't as good as I expect. I wonder there is any document about a similar implementation out there.

That's our terrain with vertex color applied to simulate ambient occlusion.

• What exactly do you want to improve? Your result seems quite well in my opinion. Ambient occlusion exactly does that, it darkens corners. But it is better to add ambient occlusion to the ambient color instead of multiplying it. By the way your game Midera looks really nice! – danijar Mar 26 '13 at 11:35
• Thanks for the compliment! And what i am trying to do is automatically do the vertex colors at runtime so that I dont have to do it in my modelling programs. I am big on trying to avoid shaders unless i need them. As some computers cant handle SSAO and doing the AO this way would allow them to see the detail shading without having a good computer. That and it would run faster on a mobile device =-) – Mungoid Mar 26 '13 at 13:24
• Why not let the user choose? Many games out there provide graphics settings where you can enable or disable several effects. And shaders are exactly the way to go for such effects. The GPU is typically much faster for this kind of parallel computations. As long as you don't want to return the result back to the CPU, you should use shaders. But I haven't understand you completely. Does the second image shows manual or computed ambient occlusion? – danijar Mar 26 '13 at 15:27
• Ultimately i do plan on having an option for more advanced shading. But between vertex shading and pixel shading, vertex is more accessable to older graphics cards and I would think faster too. And the second image is the same as the first one but I manually colored the vertices just to use as an example. However, I'm not fully experienced with shaders and GPU performance, so its very possible I could be thinking about this the wrong way – Mungoid Mar 26 '13 at 15:58

Interesting question! What you're trying to do sounds a lot like SSAO, but per-vertex rather than per-pixel. The same basic approach should work - calculating the AO factor for each vertex using a formula involving the positions and normals of the neighboring vertices.

One of the more popular ways to do SSAO is Volumetric Obscurance. Geometrically, the idea is to think of a sphere centered on each vertex, and calculate the AO based on what fraction of the sphere's volume is buried under the ground. On flat ground 50% of the sphere will be buried, so there should be no AO in that case, but if greater than 50% of the sphere is buried then there should be some proportional darkening.

The neighboring vertices can be used to estimate the buried fraction by treating each one as a line sample parallel to the central vertex's normal. You can calculate the buried portion of the line sample with a little vector math, then average this over all the neighbor vertices, weighted by how long each line sample is. Something like (untested pseudocode):

float heightTotal = 0.0;
float heightMaxTotal = 0.0;
float radius = 1.0;  // Radius of AO sphere; tune to taste

for each neighborPos:
float3 neighborOffset = neighborPos - vertexPos;
float height = dot(neighborOffset, vertexNormal);
float3 tangentProj = neighborOffset - vertexNormal * height;
float distSqr = dot(tangentProj, tangentProj);