isometric tile - How would I apply effect on only specific part of a sprite?

Question

How would I apply a mask for pixel shading effect to restrict it only to some part of image? For example I have a height tile:

Now I want to apply the effect only on the tile-shape of this height-tile with some sort of masking:

Therefore, how would I restrict the pixel shading function boundary using masking?

Answer 1

You could achieve this through (at least) two different methods. One is to use the stencil buffer, this way you could draw arbitrary shapes in pictures and use those as masks. The other option is to make the mask in a vertex buffer and thus draw only the wanted part directly in the first call.

Both would work fine, in this case the second type would probably be simplest and quickest to setup, but the first one allows a bit more artistic freedom at the cost of some complexity. Here is a pretty good example for stencil buffering and what it can do.