To apply different textures based on height the general method is to, in a pixel shader;
- Find the height of the geometry at the current pixel position.
- Perform a texture lookup on the heightmap that defines the geometry or just find the position of the current geometry as you would for lighting and find the y component.
- Supply your shader with some kind of look up table matching a threshold height to a texture.
- Index the table using your height.
- Optional - take the colour from two textures and average them using a weight when crossing a texture transition zone. This blends the transition and prevents hard edges.
Ok so that's a pretty straight forward method, but how do you apply this based on angle. Well I think the simplest way is to do exactly the above but replace height with "magnitude of the y component of the surface normal."
If you take the normal of a surface, which needs to be a unit vector (so normalise if appropriate), then the flatter the surface the greater the contribution in the y component.
- A perfectly flat surface pointing up would have the normal (0, 1, 0).
- A surface on it's size will have the normal (X, 0, Z).
- A surface pointing down will have the normal (0, -1, 0).
So make a lookup table based on that Y. You can take it a step further as well and use both the geometry position and normal Y component to do a 2 dimensional lookup. i.e. In low terrain use grass for level geometry and rock face for vertical geometry. Higher up use rocks for level geometry and frozen rocks for cliff faces.