I am currently developing a game engine that utilises signed distance fields as a rendering technique to display smooth procedural geometry (generated with simple primitives like the ones in your link for now, looking to implement Julia and IFS fractals in the future). Since my engine is focussed on procedural generation and must define figures in a way that makes them ray-marcher friendly, I figure I'm in a good place to answer this question :P.
Regarding streaming, the simple solution is to use a typed buffer of some sort and throw it onto the GPU when you want to do your ray-marching. Each element of the buffer is a complex type (e.g. a struct in C/C++), and each type contains elements defining what function you should use to represent it, it's position, rotation, scale, etc., and an average color. The process then simplifies down to:
- Cull your scene into a manageable subset (note that frustum culling and occlusion culling are partly performed automatically by the ray-marching algorithm anyways)
- Pass the subset into your render input buffer
- Pass the buffer to the GPU if it isn't there already, then render your scene with ordinary traditional ray-marching. You will need to perform some kind of per-step search to evaluate which item in the input buffer is closest to each ray for each iteration of the ray-marcher, and you'll need to apply transforms to either the rays (in which case you'll need to invert figure rotations before they reach the GPU) or the distance functions themselves (moving the function origin for position changes, adjusting e.g. cubic side lengths for scale changes, etc.) The simplest approach is to just modify the rays before you pass them along to the actual core distance function.
Regarding figure colors, remember that shaders allow you to define complex types as well as primitives ;). That allows you to throw everything into a C-style struct, then pass those structs back from your distance function.
In my engine, each struct contains a distance, a color, and an ID that ties it to the corresponding figure definition in the input buffer. Each ID is inferred from the surrounding context of the relevant distance function (since my mapping function loops through the input buffer to find the closest figure to each ray for each step, I can safely treat the value of the loop counter when each SDF is called as the figure ID for that function), while distance values are defined using an arbitrary core SDF (e.g.
point - figure.pos for a sphere), and colors are either defined from the average color of the appropriate element in the figure buffer (hence why it's useful to keep the figure ID's around) or through a procedural color weighted towards the stored average (one example might be taking an iteration count for some point on the Mandelbulb, mapping your "average color" from FP color space to integer color space, then using the mapped color as a palette by XOR'ing it against the iteration count).
Procedural textures are another approach, but I've never used them myself. iq has done quite a lot of research in that area and posted some interesting demonstrations on Shadertoy, so that could be one way to gather some extra information.
Regardless of whether your color is static for each figure, procedurally generated, or magically sampled from a procedural texture, the basic logic is the same: Abstract figures into some sort of intermediate complex type (e.g. a struct), store both local distance and local color in an instance of that type, then pass the complex type as a return value from your distance function. Depending on your implementation, the output color can then pass directly through to the screen, or follow the collision point into your lighting code.
I don't know if the above was clear enough or not, so don't worry about asking if anything doesn't make sense. I can't really give any GLSL/pixel-shading code samples since I'm working with HLSL and compute shading, but I'm happy to try and go over anything that I didn't write properly in the first place :).