I have terrain being rendered in my project using VBOs in OpenGL. I would like to apply some Frustum Culling but have no idea how to access each polygon as its drawn to check if it is in view. I think this is where octrees come into play but I have no idea how this is really done. Does anyone know a good tutorial on how to do this specific thing or have some pseudocode for me?
Thanks in advanced!
More information on how you are storing the terrain data would be helpful here. Are you storing it chunks? Are you constantly rebuilding the VBOs? Is all of the terrain in one VBO and you are rendering pieces of it at a time?
Trying to cull per polygon is probably going to introduce more overhead than it saves. The larger a chunk of terrain you can test at once, the quicker the culling will be, with the obvious trade-off of accuracy. In the end, you will need to do some profiling to figure out what the optimal point of that trade-off is, so creating a system that allows you to dynamically change how your terrain is broken up will be very helpful.
A GPU can already cull the polygons on a per-polygon basis, but this happens at rasterization/setup time which happens after vertex shading (using Vertex/Geometry/Hull/Domain shaders). This means that the GPU can still end up shading polygon vertices that will end up getting culled. If this happens, the GPU will have done a lot of work for nothing and is therefore a waste of GPU cycles.
For this reason, we try not waste GPU cycles by culling invisible geometry on the CPU side (or GPU side using DX10+ features such as DrawIndirect and/or predicated rendering) in batches of polygons by simply not calling draw for those batches.
Typically that batch of polygons can be represented by a sphere or box that surrounds all those polygons. This bounding sphere/box is then tested against the frustum and if it touches the frustum, the draw call is invoked for those polygons. If not, the draw call is skipped. An octree is simply another way of determining which batches of geometry are visible, except it organizes that data in a hierarchical fashion. Octrees are not necessarily suitable for all geometry.
Some fairly good references that explain the sphere/frustum technique follow:
