Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I'm seeking advice, suggestions, and ideas on how to handle the updating of large amounts of data in OpenGL and c++.

My partner and I have gone through two methods.

The first is vertex by vertex rendering. Right away, this was crossed off the table; it's super slow! When using it, the simulation ran at 3 FPS. All it was rendering was just some untextured cubes composing a flat landscape. Imagine adding some animals to that, or even a full blown world! This method is definitely not an option.

The second method is VBOs. The problem with VBOs is that they don't like having vertice added or removed (though removing can be accomplished with a hack of setting data to null). We could just create a new VBO every time something changes in this way, but the 3D data is likely to change often, if not every frame. Thus that would also be very inefficient, and VBOs are not an option either since, we can't add vertices/faces and can only dirty-hack remove existing vertices/faces.

What, then, would be the best method for us to use? Would we be able to use a shader or something to create a custom datatype on the graphics card, upload the 3D data to it initially, and then afterwards, only send the transformation details (such as a transformation matrix or what have you) and what faces/vertice have been removed/added? Even if the above method is a possibility, how else could I go about doing this, and what is its pros and cons? Perhaps one of those would better serve my needs.

Edit: Here is what I mean by "large amounts of data". It shall be known that the project that requires this fast updating of "large amounts" of OpenGL data is a simulator that attempts to simulate its own world. World being defined as an entire existence, which may include multiple universes, dimensions, galaxies, solar systems, planets, etc, such that it is not limited to a single "planet", which is a common definition of the term "world". Thus, this "large amount of data" is the portion of the world that is visible to the player.

As this is a world simulator, all the data of the world is procedurally generated. This means 3D models are NOT being loaded. The "3d models", if you wish to call them that, are generated from the 3D data of the objects in the world, such as a house, tree, car, flower, cat, or cup.

In short, the world simulation aims to simulate the world as detailed as possible (the limit being set by the resources of the computer running it). For the purpose of this question, let us assume that the simulation is being run on a computer that allows for every object to possess very high polygon counts which we will say, for the purpose of this question, exceeds 100k for a cat.

That in itself is not a problem; just initially place the data in a VBO. However, what happens as the cat grows? New vertices and faces would be added and removed. The case extends to actions the cat does such as walking around the world (Animations are not explicitly defined and used in the code. They are, rather, a byproduct of the simulation, in which the 3D data itself is updated, creating the effect of an animation). That is where the problem lays. Creating a new VBO for every frame of the 'animation' (keep in mind the above statement on animations), or for every time the cat grows, is not an option due to performance concerns. It'd more or less achieve the same results as vertices by vertices rendering, but with more overhead, and the only performance gains seen being when no new vertices and faces are added or removed.

In conclusion, large amounts of data, such as 100k polygons per model, will be seen. The specific edits that will be done to this data results from the growing, or otherwise shrinking, of objects in the world and the simulation of these objects (See the note I made earlier on animations.)

Edit2: To clarify confusion, this is how the data is changed:
1. The position of each vertices is updated relative to the camera. Eg, a dog walking away from the camera.
2. Vertices are added and removed from the model. Thus faces are also changed, removed, or created.

Changes #1 and #2 are expected to occur at least once a second for every object in the world, excluding terrain, rocks and other inanimate objects.
I do not see how patterns of changes can be predicted, as the data can change in unimaginable ways.
Examples of changes that will occur:
1. The simplest case, an apple falling from the sky. This is easily be done by simply applying a transformation matrice. The rest that follow are more complex.
2. A sword lopping off the arm of a soldier. In this example, vertices are removed from one model and put into a new model. VBOs do not explicitly support this, but a dirty hack exists to work around it.
3. An engineer welding two pieces of metal together. In this example, vertices are removed from one model, and added to the next. VBOs do not support this. In this specific case, that problem can be worked around by simply retaining the individual VBOs and only do the merger of data in the world simulation code.
4. A chemical reaction. Such examples of this would be an acid eating away a material or mixing liquid detergent and hydrogen peroxide reference video)
5. A physical reaction, such as ice melting into water.
6. Liquids. The most complex case, as their surface dynamically changes in countless ways.

Edit 3:
A wave of inspiration swept over me and I've come up with two ideas on how to solve this problem. I'm now working with my partner towards implementing both. In short, one method creates a custom data structure similar to VBOs, that has the in-built feature of adding and removing vertices, while the other is a hack using VBOs, destroying the old one whenever vertices are added or removed and recreating it, but doing it on the GPU so that data is not transferred all the time, which is the source of the nightmarish slow performance of vertex by vertex rendering. More details to come.

share|improve this question
1  
Can you edit your question to quantify your "large amount" of data, and to be more specific about what edits you plan to make to it? Right now it sounds like a VBO is inappropriate for what you're trying to do, but your details may make all the difference. –  Seth Battin Aug 21 '12 at 22:29
    
I have done as requested. –  Addictgamer Aug 21 '12 at 23:00
1  
That would only be a problem/occurrence if the simulator is simulating our Earth, or if somebody (including one of the entities in the world) decides to program their own simulator. Regardless, I'm not going to worry about that. By the time the world simulation progresses to such a state where players will be writing their own world simulations or the entities therein do such, computers will be powerful enough to handle it. See: Moore's law, and 2050 Singularity. For now, the problem on hand is the fast rendering of large amounts of OpenGL data, as detailed in the question. –  Addictgamer Aug 21 '12 at 23:49
2  
This question is too poorly-defined and nebulous to be answered. You're basically saying, "I have a lot of arbitrary data that changes arbitrarily. How do I reflect that in GPU memory for maximum performance?" Exactly how data changes is important to devising a strategy for uploading it reasonably fast. Without knowing anything about the patterns of the changes, what will become what, or any question of limits, there's no chance for any actual answer. –  Nicol Bolas Aug 22 '12 at 5:51
1  
@Addictgamer , are you trying to do the same as they did in that movie called The 13th floor ? It seems you're cramming up tons of things into a very, very impotent system (regarding this job) as a today's 3D-related programming languages. Statistically, there's no such simulator.. people are working only on tiny details: e.g. the 4th and the 5th points of your last paragraph are themselves HUGE research areas (see the SIGGRAPH papers for instance). Ask this: what data-structure can I use that will support topology changes. That's what you need for the intimidating endeavor. Best of luck! –  teodron Aug 22 '12 at 7:46
show 5 more comments

1 Answer

You have a couple different problems here.

First, the transformation of the data and getting those changes to the GPU. If at all possible, you should aim for moving the transformations themselves to the GPU. This might very well require using OpenCL/DirectCompute/CUDA, as the standard shader stages don't really do what you need for most non-trivial transformations. In general, if you're trying to process hundreds of thousands of vertices moving around, the GPU is going to do a better job than the CPU, unless your particular deformation algorithms happen to be one of the (many) sorts of algorithms that don't work well on GPUs. You haven't provided enough details to tell.

Second, you have lots of vertices on your models. A problem that you're assuming you have -- but which you probably don't -- is how to draw a 100k vertex model. The question you should be asking is whether you need to draw a 100k vertex model. At a few dozen meters distance, odds are that model is going to be indistinguishable from a simplified 10k vertex model. At a few hundred meters, it'll look the same as a simplified 1k model. At large distances, it'll look just fine as a billboard sprite. This is basic LOD (level of detail) work for a standard pre-made 3D model. For your procedural models, it may be a bit trickier; however, it's still almost certainly possible, and if you're worried about rendering performance and GPU bandwidth is absolutely the path you want to go down.

Second part deux the third, you possibly have a lot of models if this is a complex simulation. Just as you want to reduce the visual complexity of your models to avoid processing things the user can't see, you're going to want to use visibility culling to avoid processing entire models (or sub-sections of models) the user can't see. If you've got a dozen 100k models but they're all behind the camera, there's no reason to even think about how to render them, so just don't. There are many great data structures for spatial scene management to help with this. Likewise, for large and complex models, you can use similar (or even the same) data structures to split your models up and avoid any rendering overhead for parts that aren't visible. If you have a large cat model that is spatially separable between its head, body, legs, and tail, and the user is zoomed up close on the cat's head, you may be able to determine that the legs are not visible at all and thus simply not need to worry about drawing them. This works well with the previous paragraph, since you'll likely either be zoomed in to a single object (and you can cull most everything else) or you'll be looking at a larger scene where all the objects are smaller in perspective (and hence you only need to render simplified versions of them).

The specific algorithms that are available and which are the best to use for your specific data and needs will vary. If your meshes are very deformable then you'll probably want to use a very different algorithm than if they only scale. A good first cut might be basic kd-trees. Going from there, you should look into the research done on voxel engines (real voxel engines, not the Minecraft clones) as they generally deal with the same kind of high-detail models you are. I am unfortunately not versed in that topic nor do I have any good resources to point you to, but maybe knowing what to look for will help you find an answer. (And if you do find one elsewhere, please post it here!)

share|improve this answer
    
1. Optimally, I'd have the graphics code take the object's data (as stored in the simulation), and go through all of its points and render the visible ones. The problem with that is vertex by vertex rendering is super slow. To work around this, the simulation, in the current design, gives the object's data initially to the graphics code, and afterwards only sends the graphics code a list of transformations, transformation matrix, whatever is deemed as best in this situation. You suggest that the transformation matrice/information be passed to a function running on the GPU which updates ... –  Addictgamer Aug 22 '12 at 7:24
    
the data accordingly? (Currently we seem to be leaning towards using transformation matrices to specify transformations, though this will change if some other method is deemed better for our use.) –  Addictgamer Aug 22 '12 at 7:25
    
2. I've already thought of this and come up with several methods of combatting this issue. Sorry that I was not clear in my original post. - 1. "Simulation Data Depth", which specifies how detailed to make (and simulate) the objects, and thus the 3D models generated. - 2. LOD, as you have suggested. - 3. LOD is very easy and simple to implement with my procedurally generated models due to how I'm generating them. It's a simple case of limiting recursions as distance increases. This is due to what I call the "Recursive Map System", ... –  Addictgamer Aug 22 '12 at 7:25
    
... which is similar to octrees. The only problem is I'd have to regenerate the model everytime the object's LOD changes. This won't be an issue in a decade, when computers are many times more powerful, and it may not be an issue on my GTX 580, but for the average computer, it is. That is why "Simulation Data Depth" can be set by the user to something low. LOD won't need to change for a while, and when it does, the models are much detailed to start off with anyway, to the point where it may look like Minecraft. –  Addictgamer Aug 22 '12 at 7:27
    
3. Yes, it is a compelx simulation. There will be a lot of models. Culling has definetely been planned :) No use rendering the underground cavern the camera can't see, or the lion sneaking up on the player from behind, or the other such examples. Exactly how the culling is implemented, I leave up to my partner, whom is writing the graphics code. I'm pretty sure he's already aware of such things, as he has prior experience with 3D programming. However, I'll make sure to go over spatial scene management, and culling in general, with him. Your suggestion to spatially split up the model - ... –  Addictgamer Aug 22 '12 at 7:27
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.