We have a deformable terrain surface. Initially ambient occlusion was baked, but since the terrain can deform in real time, this approach isn't practical anymore. Therefore we used vertex paint to imitate occlusion based on the angle of the surrounding terrain vertices. We will then multiply the vertex color with the terrain texture and get a relatively cheap ambient occlusion.
Below are two examples of our terrain. The second has applied occlusion based on surrounding vertices, as mentioned before. But unfortunately it seems like the lower a set of vertices is in relation to the surrounding vertices (or the tighter the crease), the darker the vertex will be.
Vertex lighting in general works, but the results aren't as good as I expect. I wonder there is any document about a similar implementation out there.
That's our terrain without additional shading.
That's our terrain with vertex color applied to simulate ambient occlusion.