Recently I've been working on a realistically scaled planet engine. I found several interesting links about how to do it, what problems will come up after, and how to solve them. The approach I took was the one I found on many articles: Use 6 quadtrees, one for each faces, then transform the cube into a sphere.
My quadtree logic seems to be working, I tried it on a flat terrain and on a sphere ( with reasonable distances ), and it worked great, then I tried with bigger distances.
The first problem was z-fighting, fixed using a logarithmic depth buffer ( http://outerra.blogspot.fr/2009/08/logarithmic-z-buffer.html ). The second was floating point precision on big numbers.
I read two main approaches to fix it. The first one is to use doubles instead of floats for operations, and give the gpu every positions relative to camera space. The second was to use different spaces in different units ( object space in meters, planets space in kilometers, solar system space in AU, etc... ). While the second was encouraged by many people, I took the first one because it seemed easier to do. I plan to try the second in the future anyway.
The first approach worked on flat terrains because it was easy to compute the position relative to the camera ( centerOfNode - cameraPos ). One bug I still have with it is about height. I use 3D simplex noise on the gpu, which works in float and needs a vec3 input. With relatively small distances, the worldspace position works great. With huge distances, the worldspace position doesn't work when far from the origin, the pattern is very odd because of missing precision. And of course, it won't work with cameraspace position as input. I haven't find anything about that. Something I will try tomorrow is the second approach above. That is, define several spaces with different units, transform the position in each of the spaces, and use each of them as input for the simplex noise function, which will give me several textures I will be able to use for scale. I don't know how I will do in details, but I think it won't work since transform from the lower to the high space might result in floating points errors right? I could dynamically create vertices in the CPU using simplex noise with double precision but it doesn't sound right... I'd like to use the GPU as much as possible, unless I have no other choices.
Second problem I have is with spherical terrains this time. So my steps to get the planet are:
1) From the quadtree nodes, scale, rotate and translate an unit quad to his position in the current plane of the 6 quadtrees
2) normalize it to get the normal
3) multiply the radius to get a sphere at sea level
4) add the height
Since the first step is related to the origin of the planet, even in view space it will require operations with huge numbers ( the radius ), so I am not sure how I can avoid precision problems here... Again, I could create them on the CPU and send them to the GPU, but this sounds exhausting and I don't want to do it unless I have no other reasonable choices. I would gladly use the second solution ( several spaces ) if I had an idea of how to fix it with it, but I'm not sure how to do it...