Performance of glDrawElements vs glDrawRangeElements vs glDrawArrays

Question

I've looked around on the web trying to find a good answer to this, but am unable to find one. Different forums mention driver bugs and the likes, but these discussions are usually almost 10 years old, so bugs should probably have been fixed by now. So, to the question!

I'm drawing a mesh that consists of 628 008 vertices, normals and texture coordinates. I also have indices for the mesh, but the indices are the worst kind of indices: they do not re-use vertices, but actually just list the vertices in order (or close to). So we have 628 008 indices as well. Yes, I can re-write this to re-use vertices, but that's not really where my question is going.

The mesh is drawn using VBOs (vertices, normals, texture coords) in a VAO, and two different shaders: one for texturing one part of the mesh, and another shader for another non-textured part. To do this, I'm using the OBJ format and material groups (exported from Blender). The mesh is split into material groups, describing with which material a batch of indices shall be drawn.

The VBOs are set up to be drawn as GL_STATIC_DRAW since it is never changed.

I then loop over the material groups and draw the range of indices described in each group, like:

  for( const auto& group : mesh)
    if( group should use texturing )
    {
      enable texturing shader with phong material
      set texturing uniforms
      set material stuff
      set mvp, etc
    }
    else
    {
      enable phong material shader
      set material stuff
      set mvp, etc
    }
    draw(group)
  }

There are 13 different groups, so the draw command will be called 13 times.

I've tried drawing the mesh in five different ways and the results are quite surprising.

1. glDrawArrays.
2. glDrawElements with indices fed using pointer to index array. (reffered to as CPU-side indices below)
3. glDrawElements using GL_ELEMENT_ARRAY_BUFFER and offset fed to command
4. glDrawRangeElements with indices fed using pointer to index array.
5. glDrawRangeElements using GL_ELEMENT_ARRAY_BUFFER and offset fed to command

The [start, end] vertex indices fed to glDrawRangeElements are calculated when reading the mesh at start-up so they are correct and precise. This could be a performance problem otherwise, according to the spec.

The results in ms (measured on CPU side using C++11 timers, using 500 samples and calculated mean value, collecting 3 values since the first frame's timing is huge):

glDrawArrays:
            Median: 8.0 Mean: 8.9 Max: 217.0 Total: 4285.0 Samples: 482
            Median: 8.0 Mean: 8.8 Max: 43.0 Total: 4345.0 Samples: 494
            Median: 8.0 Mean: 8.2 Max: 14.0 Total: 4125.0 Samples: 501

glDrawElements:
GPU Indices (ELEMENT_ARRAY): 
            Median: 34.0 Mean: 44.1 Max: 597.0 Total: 20388.0 Samples: 462
            Median: 34.0 Mean: 33.9 Max: 63.0 Total: 16973.0 Samples: 500
            Median: 33.0 Mean: 33.5 Max: 80.0 Total: 16797.0 Samples: 501
CPU Indices (pointer to CPU-side indices):
            Median: 9.0 Mean: 10.3 Max: 236.0 Total: 5027.0 Samples: 488
            Median: 9.0 Mean: 9.2 Max: 32.0 Total: 4581.0 Samples: 500

glDrawRangeElements:
CPU Indices (pointer to CPU-side indices):
            Median: 9.0 Mean: 10.1 Max: 221.0 Total: 4640.0 Samples: 459
            Median: 10.0 Mean: 9.6 Max: 22.0 Total: 4780.0 Samples: 500
            Median: 10.0 Mean: 9.5 Max: 32.0 Total: 4774.0 Samples: 501

GPU Indices (ELEMENT_ARRAY): 
            Median: 35.0 Mean: 36.9 Max: 597.0 Total: 16101.0 Samples: 436
            Median: 34.0 Mean: 35.0 Max: 55.0 Total: 17517.0 Samples: 500
            Median: 39.0 Mean: 39.1 Max: 64.0 Total: 19448.0 Samples: 497

So my huge surprise is: How come GPU-side indices takes so much longer? The frame rate literally drops to less than half (from about 34 fps to 15 fps).

What can be the reason for this? My guess would have been that the performance timings should be in this order (since we have crappy indices and they aren't really re-using anything):

1. glDrawArrays <- Fastest
2. glDrawRangeElements (using ELEMENT_ARRAY)
3. glDrawElements (using ELEMENT_ARRAY)
4. glDrawRangeElements (using pointer to indices)
5. glDrawElements (using pointer to indices) <- Slowest

Why do you think that the performance is so bad using ELEMENT_ARRAY_BUFFER bound to the VAO? Is this driver specific? Is it because there are so many indices and memory have to be swapped around in the driver?

Hardware: MacPro running Windows 8 Dual AMD FirePro D300, 2048 MB dedicated graphics memory

Answer 1

I would expect glDrawArrays to be fastest on both the GPU and CPU for this scenario, primarily because you're saving the GPU memory bandwidth otherwise used for fetching the index data and the CPU cost of managing the index buffer.

I do not expect glDrawRangeElements to be any faster than glDrawElements - in fact, I would expect it to be slightly slower. The range passed to the function is intended to be used to allow a driver to know which sections of the vertex data need to be made available to the GPU. However, if you're using a vertex buffer, this is not useful information. Further, it's also just a hint - drivers are expected (although technically not required) to render correctly even if the range is wrong, and it's surprising how many applications do get it wrong. So, we generally ignore those parameters.

As for why glDrawElements is faster with client index data, that's odd, but really depends on the sequence of calls. You say your overall mesh has 600K+ vertices, but it's broken into groups. What is the size of each group? Each time you switch states (such as binding a new index buffer), the driver needs to validate the contents of the buffer, which takes time. If you're rapidly switching between two states, then this could be where your CPU time is going. When you use an index buffer, each switch is (potentially) a new buffer and is more work for the driver to validate. If you use client indices, even though the driver might be transferring indices to the GPU, it's all technically the same buffer - a gigantic buffer that starts at CPU virtual address 0.

Of course, it's also possible that the driver doing something silly.

Ultimately, 40 FPS is pretty low for drawing 600K vertices. Something bad is happening.

Here are a few things you can try to improve performance:

1) If your fragment shader and fixed function state is basically order independent (depth test on, stencil and blend configured for order independence), sort the groups in the mesh by textured vs. untextured, then draw all of one set followed by all of the other with no state changes in between. You can even sort further - by texture or other parameters, for example.

2) If any state is common between groups (mvp matrix, for example), make sure you're not setting it redundantly.

3) Make sure all groups in the mesh are using the same vertex and index buffers, ideally bound into the same VAO. When you draw, you can use the {first} or {baseVertex} parameters to offset into a larger vertex buffer. You can even use the same buffer object for both index and vertex data - just subdivide it.

4) As a last resort, you could try making all groups use the same shader by simply binding a dummy texture with an opaque white pixel into it for the untextured groups. If you're really CPU bound here, the extra GPU cost of fetching the (unneeded) texture data and a few extra shader instructions probably won't matter.

Once you do get the glDrawElements path going as fast or faster then the glDrawArrays path, do consider optimizing the vertices for reuse. A triangle soup mesh with perfect reuse can perform up to three times better than the same mesh rendered with no reuse.