I wrote a simple UI system for my game. There is a clip flag on my widgets that you can use to tell a widget to clip any children that try to draw outside their parent's box (for scrollboxes for example). The clip flag uses glScissor, which is fed an axis aligned rectangle. I just added arbitrary rotation and transformations to my widgets, so I can rotate or scale them however I want. Unfortunately, this breaks the scissor that I was using as now my clip rectangle might not be axis aligned.

There are two ways I can think of to fix this: either by using the stencil buffer to define the drawable area, or by having a wrapper function around my sprite drawing function that will adjust the vertices and texture coords of the sprites being drawn based on the clipper on the top of a clipper stack. Of course, there may also be other options I can't think of (something fancy with shaders possibly?). I'm not sure which way to go at the moment.

Changing the implementation of my scissor functions to use the stencil buffer probably requires the smallest change, but I'm not sure how much overhead that has compared to the coordinate adjusting or if the performance difference is even worth considering.

  • 3
    \$\begingroup\$ Or you can do an intermediate render-to-texture of your control with scissoring enabled and then draw the resulting texture with the corresponding rotation to the screen. \$\endgroup\$
    – r2d2rigo
    Mar 25, 2012 at 5:03
  • \$\begingroup\$ @r2d2rigo that's the way to go IMHO. You can even ditch the scissoring and just use masks+shader. In fact, I would cache the resulting texture as long as it's valid. If I were you, I'd post that as an answer and get my well-deserved reputation. \$\endgroup\$
    – kaoD
    Mar 25, 2012 at 16:24
  • \$\begingroup\$ If all of the componenets can be rotated in arbitrary ways, then you can't just render all elements are rotate the final render target. However the intermediate render to texture is a good idea for menus in general. \$\endgroup\$
    – Pod
    Mar 25, 2012 at 17:52
  • \$\begingroup\$ @Pod, yep, I think r2d2 meant several render targets (one per UI element, or at least per transform-group) and then rendering the final (pooled) textures with any relevant transforms applied. \$\endgroup\$
    – kaoD
    Mar 25, 2012 at 18:03

3 Answers 3


Since your sprites can rotate, clipping them manually won't be as easy as just moving your vertices and texture coordinates around. There are many cases where you'd actually need to replace a single vertex with two new vertices. But by doing so your primitive would stop being a triangle so you'd also need to triangulate the result, which is not simple. Check the following image for a clearer view:

enter image description here

So in the end, that entire processing might even be more expensive than using the stencil buffer (haven't tested though). So my advice for now would be to try using the stencil buffer first, and if it turns out to work well enough, don't give it a second thought.


Of course, there may also be other options I can't think of (something fancy with shaders possibly?). I'm not sure which way to go at the moment.

I hope I can help here, however Disclaimer: I don't know much about OpenGL, if anything. In DirectX I'd do this as a a user clip plane (dx9) or a ClipDistance (dx10+). They're both identical concepts and I know that OpenGL has identical concepts, however I'm not sure what they are so I'm just gooling randomly to fill this answer out. Hopefully this answer helps you start looking for them :)

If the version of OpenGL/GLSL you're targeting supports gl_ClipDistance, you could use those. You need to use 4 planes to define the box that you previously used a scissor for. The clip planes don't even have to be axis aligned, which is a plus point. Hopefully you can just define them to be the same as your previous scissor-rect and then just use the same rotations you did to the gui element to orientat the rects to be exactly the same?

Once you have 4 clips planes describing your clip-box you need to pass them into the vertex shader as 4 vec4 uniforms. You then find the distance between your vec4 position value and the 4 planes. (The clip plane coordinates will obviously need to be in world or clip space depending upon where you do the calculation in regards to your matrix transforms, if you even have any for gui elements. Infact if you do the gui-element's rotation in the vertex shader, you could even do the clip-plane's rotation here using the same rotation matrix?)

IIRC the distance between a point and a plane is just a dot product? Assuming it is, the 4 dot products will give you 4 scalar values, and these are the distances from the 4 planes, aka the 'clip distance'. The HW clips any edge that has an interpolated value <0. So if you have 2 verts, A and B, with a single per-Vertex clipDistance element whose value is -1.0f at A and 1.0f at B, then the HW automagically clips it for you at new position C, which will be halfway between the two :) David Gouveia's diagram above has a nice diagram reflecting this.

Hopefully you can output 4 clipplanes (the documentation refers to gl_MaxClipDistances, but I don't know how to look up a standard value for this. I hope you do :)). If not you can just accumulate the previous values, remembering that any value <0 would result in clipping.

There's also glClipPlane. That might be fixed function -- again, I know nothing about OpenGL. But it basically does the dotproduct for you and outputs the clipDistance itself, I hope. At least that's how the analgous DX functions work. SetClipPlane ClipDistance


I have done a stencil approach to that problem and runs fine...

Is done with Xna but it should be done in a similar way with opengl...

Clipping polygons in XNA with stencil (not using spritebatch)



You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .