1
\$\begingroup\$

I'm trying to reproduce color picking on iOS, following the technique described here: http://www.lighthouse3d.com/opengl/picking/index.php3?openglway2, with OpenGL ES 2.0.

When the user touches the screen, I need to draw the objects on a buffer different from the one displayed on the screen, which seems to be done by glRenderMode(GL_SELECT); how could i do the same with OpenGL ES 2.0?

Should I create a support buffer as described here? https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/WorkingwithEAGLContexts/WorkingwithEAGLContexts.html#//apple_ref/doc/uid/TP40008793-CH103-SW6

Thanks for any suggestion,

DAN

\$\endgroup\$
1
  • 1
    \$\begingroup\$ I now realize that you answered you own question. Of course AFTER writing a book on it below. So I hope you'll consider my answer as being a more complete answer. :) \$\endgroup\$
    – badweasel
    May 20 '14 at 8:53
2
\$\begingroup\$

Yes you can do it that way. This is how I do color picking in iOS and it works really great. Also, I've ported my code to android using apportable and the color picker works on other devices as well.

  1. Create a scratch or temp framebuffer. I create a specific color picker frame buffer. It's just like creating a dynamic texture. It's done the same way. I pick a buffer size that is square and large enough to fit the whole screen. You don't have to do it that way, but that's how I do it. Alternatively you could do one that's half size and the picker sample resolution might be good enough. You have to play with that for your app. If you need code let me know.
  2. Attach a depth buffer if you need to. I do only if my 3d objects I'm picking need it.
  3. Each item that you want to be pickable needs a unique color:

    -(vec4) colorVec4FromItemID: (int)itemID
    {
        if (itemID<0) {
            return vec4Make(0.0, 0.0, 0.0, 1.0);
        }
        float r = itemID % 255;
        float g = (itemID / 255) % 255;
        float b = (itemID / (255 * 255)) % 255;
        return vec4Make(r/255, g/255, b/255, 1.0);
    }
    
  4. I track items one of two ways. Either I have my objects in a struct array and the itemID I pass to the above method is the index number of that array. Or I keep a colorPickerID and ++ it each time I grab a color. Either way I store the color in a vec4 pickerColor in the array of struct. I avoid objective-c ns-stuff for color picking and use a c array of structs. In my games I also multiply my input id by 30 to give some space between the colors - but this probably isn't needed.

  5. Render using a solid color shader. So I re-renader the pickable objects in my scene to the color picker frame buffer/texture when needed (Not every frame). I have blending off for as hard an edge as I can get. You don't want interpolated colors. And I use a shader like this:

     void main(void)
    {
        gl_FragColor = u_baseColor;
    }
    
  6. That also means that I keep my shaders and support code very compatible with each other. I can flip to a solid color shader from a texture shader without having to redo a bunch of code. Then you can reuse the same draw routines for both passes. And when drawing the color picker pass I also use the exact same matrices. And if you cache them you don't have to recalc them.

  7. When the user touches that's when I set processColorPicker=TRUE. In my drawView loop AFTER drawing my scene I check that bool and then (if the scene has changed) I draw the color picker and process it. Obviously don't draw the picker each frame.

  8. Read the pixel:

    -(void)processColorPicker
    {
        //if (colorPickerNeedsRedrawn) {
        [self drawColorPicker];
        //}
         glBindFramebuffer(GL_FRAMEBUFFER, pickerFramebuffer);
         glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureInfo[TEX_PICKER].texture, 0);
         Byte pixelColor[4] = {0,0,0,0};
         glReadPixels(colorPickerTouchPosition.x * backingScale, (backingHeight - (colorPickerTouchPosition.y * backingScale)), 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixelColor);
         vec4 pickerColor = vec4Make((float)pixelColor[0]/255, (float)pixelColor[1]/255, (float)pixelColor[2]/255, (float)pixelColor[3]/255);
         NSLog(@"what is pixelColor at %f,%f = %f %f %f %f", colorPickerTouchPosition.x, colorPickerTouchPosition.y, pickerColor.r,pickerColor.g,pickerColor.b,pickerColor.a);
        ...
    
  9. Interpret the read. You can see if the color matches a picker color stored in your struct or you can check each id using the colorVecFromItemID again for each item number.

Some tips: If you keep only reading 0,0,0,0 and it's not working chances are either your buffer is set up wrong or you're not really drawing to that buffer. You could always draw the texture to the screen instead of processing the color.. then you can see if the colors are drawing correctly. They'll be dark and reddish.

Also, since you're not reading from the screen but from a scratch buffer you don't have to worry about kEAGLDrawablePropertyRetainedBacking. That only matters if you're reading pixels from the screen buffer. Also I'm not sure Brad Larson's alternative to glRead will work for this either since I believe that only works reading from the screen.

IF you want to speed things up.. divide the size of the color picker texture by 2 or 4. The buffer will theoretically draw 4 times faster at /2 and 16 times faster at /4. Chances are you don't need a retina pixel accurate color picker buffer.

Make sure you don't start drawing to the picker buffer (or any dynamic texture) in the middle of when you're drawing to the main screen buffer. The way I do that is to not jump directly into processColorPicker but instead to set a bool and then do it in my drawView loop after the main buffer has been presented.

* UPDATE **

Here's my color picker buffer code. Note that I keep information on all my textures in a textureInfo array, including if it's been generated, the size, etc.

-(BOOL)createPickerFramebuffer
{
    glGenFramebuffers(1, &pickerFramebuffer);
     glBindFramebuffer(GL_FRAMEBUFFER, pickerFramebuffer);

     glGenTextures(1, &textureInfo[TEX_PICKER].texture);
     textureInfo[TEX_PICKER].generated = YES;

     glBindTexture(GL_TEXTURE_2D, textureInfo[TEX_PICKER].texture);

     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);

     glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, backingWidth, backingHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);

     textureInfo[TEX_PICKER].width = backingWidth;
     textureInfo[TEX_PICKER].height = backingHeight;

    // bind renderbuffer and create a 16-bit depth buffer
    // width and height of renderbuffer = width and height of
    // the texture
     glGenRenderbuffers(1, &pickerDepthRenderbuffer);
     glBindRenderbuffer(GL_RENDERBUFFER, pickerDepthRenderbuffer);
     glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, backingWidth, backingHeight);

    // bind the framebuffer

    // specify texture as color attachment
     glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureInfo[TEX_PICKER].texture, 0);

    // specify depth_renderbufer as depth attachment - only if you need it
     glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, pickerDepthRenderbuffer);

    // test the framebuffer
     GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
     if(status != GL_FRAMEBUFFER_COMPLETE) {
         NSLog(@"***************************failed to make complete picker buffer object %x", status);
         return NO;
    }
    //else {
   //   NSLog(@"*************************** MADE complete picker buffer object %x", status);
    //}

     glBindTexture(GL_TEXTURE_2D, 0);
     glBindFramebuffer(GL_FRAMEBUFFER, 0);

     return YES;
}

And here's how you start drawing to it:

-(void) startDrawingPickerTexture
{
    // set the framebuffer to the textureFramebuffer
      glBindFramebuffer(GL_FRAMEBUFFER, pickerFramebuffer);
      glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureInfo[TEX_PICKER].texture, 0);

    // reset the viewport just in case
      glViewport(0, 0, backingWidth, backingHeight);

      glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
      glClear(GL_COLOR_BUFFER_BIT); // and GL_DEPTH_BUFFER_BIT if necessary

      glDisable(GL_DEPTH_TEST);
      glEnable(GL_TEXTURE_2D);

      glBindTexture(GL_TEXTURE_2D, 0);
    // And now it's ready and you can start rendering to it....
}
\$\endgroup\$
8
  • \$\begingroup\$ You can improve this a lot. First of all color picking only really makes sense when your scene is somewhat static, otherwise other methods (ray collision through some tree structure) will outperform color picking very quickly. If your scene doesn't change a lot, you can cache the picking texture in RAM and get O(1) results. To save on bandwidth a bit, you may want to make the texture's dimensions half as big. You can also pick the position on the object by layering this approach. Layer 1: draw objects with unique color and pick the object. Layer 2: draw the picked object only with a gradient. \$\endgroup\$
    – bogglez
    May 20 '14 at 9:05
  • \$\begingroup\$ Thanks for the info. I've gotten really good results with this on the iPhone. Sometimes no reduction in fps. I'd like to know how to do the RAM cache in iOS. I do only redraw it when the scene has changed. I left it commented out above. I also did mention the 1/2 or even 1/4th method. I've never needed to pick a position on an object but that's good to know. Thanks. \$\endgroup\$
    – badweasel
    May 20 '14 at 9:13
  • \$\begingroup\$ Basically as long as the camera and collidible objects don't move the collision information doesn't change, so it would be a waste to rerender the buffer. So first of all only rerender the picking buffer when at least one of the two happens.Then you could just read the result to an array with glReadPixels for example (although that's slow because of synchronization, you will only need to do it once). if(collidiblesMoved) { rerenderPickingBuffer(); readPickingTexture(myArray); } pickedObject = getObjectIDFromColor(myArray[y][x]); Double buffers are useful here \$\endgroup\$
    – bogglez
    May 20 '14 at 9:21
  • \$\begingroup\$ Like I said, I don't re-render the color picker if touchable things haven't moved. I don't use it for collision detection so I'm not sure what you mean there. Depends on the game I suppose. And I haven't had problems with glReadPixels being slow when reading a dynamic buffer (not the screen buffer). I'm not sure if synchronization is an issue on a buffer that's not being presented to the screen. I'd have to test but I'd also suspect that glReading the whole texture into ram would take a lot longer than reading just that one pixel. \$\endgroup\$
    – badweasel
    May 20 '14 at 9:27
  • \$\begingroup\$ Notice the OP said "when the user touches the screen" which is the context we're going on. But I did just have a thought. I'd have to test to see if there's a speed boost. But you could turn on glScissor to only draw the area around the touch point, since that's the only area we're sampling with glRead. \$\endgroup\$
    – badweasel
    May 20 '14 at 9:30
1
\$\begingroup\$

The answer is Yes.

Creating a temporary framebuffer is the solution and Apple's documentation is just fine.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.