In order to create a plausible solar system, make sure every orbit is within the sphere of influence of the parent body, but not within the hill sphere or roche limit of another body.
The sphere of influence is the maximum radius around a planet where stable satellites can be expected.
The roche limit is the minimum orbital radius one celestial body can have around another. When it is on a lower orbit, it breaks apart and becomes a ring.
The hill sphere is relevant when you want to prevent to create two satellites around the same body which have very close orbits. It is the range between minimum and maximum orbital radius a planet "occupies".
All three values can be calculated from the mass and orbital radius with the formulas in the linked Wikipedia articles.
So I would then try the following algorithm:
- Create a random number of celestial bodies with a random orbital radius and mass. Radius and mass should be on a logarithmic scale.
- Starting from most to least massive, calculate the hill sphere of each planet. Any less massive planet in the hill sphere of a more massive planet becomes a moon of that planet. Randomly-generate the orbital radius of the moon around the parent with a logarithmic distribution between 0 and the sphere of influence of the parent.
- Perform step 2 for all moon systems to resolve hill-sphere conflicts of moons. Whether a moon can have a stable satellite is a matter of debate among the astronomy community (no example is known in our solar system). When you don't want any moon-moons, simply delete the smaller moon or put it on a different random orbit.
- Check the Roche limit of every object around its parent. When it is below the roche limit, convert it to a ring (or just delete it).
This covers single-star systems, but not binary star systems. A binary star system has two stars which orbit a common barycenter. Planets can either orbit one of the stars (S-type orbit) or the common barycenter on a very wide orbit (P-type orbit).
If you want a binary star system, I would recommend to generate the second star as another satellite around the primary star at first. Anything in the hill sphere of the second star orbits the second star and anything with a radius smaller than the hill sphere of the second star orbits the first star. Calculate the barycenter and have both stars with their moons orbit that. Anything with a larger obit than the hill sphere orbits the barycenter of the two stars (P-type orbit).
Trinary and larger n-ary star systems are only stable when the stars beyond the 2nd are very small compared to the other. These additional stars should be handled just like any other planet.