Few years ago I tried to make a simple 3D game - billiards. Completed like 50%, stuck with physics. Basically, I only need to calculate balls rolling over flat surface, but it would be nice to make something more flexible. I know all the formulas and laws (most of them, anyway). the problem is I have no idea of how to make good physics engine architecture-wise. I tried google and other forums but didn't find what I was looking for. The only suggestion was to look at open-source engine, but I'm not that good a programmer to make heads or tails out of it...
closed as not a real question by Josh Petrie, Nicol Bolas, Tetrad♦ Mar 27 '12 at 16:56
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, see the FAQ.
How to start an Architecture/Design Task: With pen and paper.
Get yourself a large sheet of paper and start drawing out the components and items that will exist within your engine. What properties each entity will need so that you can model the physical interaction.
You'll need to figure out what entities your code will have to deal with, which entity has the responsibility of running the physics simulation (and which bit of the simulation).
Then you'll have to figure out how the entities are interacting. Use colours.
Now go to sleep. leave the design for a few days and do other stuff. If you have some friends you can talk to, this is a good time to ask them about what they think.
Return to you design. You'll see some obvious problems and will feel the urge to make it more elegant. Do it.
Start coding from your second design. Rinse and repeat until you feel proud of your design and implementation. Then ask for criticism from people whose technical skills you respect, and realise that you've barely scratched the surface and that your engine is held together with duct tape. Rinse and repeat.
That last point is most important. When you get stuck, try to explain what your engine should do to a friend, on IRC, or in an email to someone whose work you respect. The act of trying to explain it to someone else will often reveal what your problem was in the first place. (You don't actually have to hit the "send" button, if you feel shy...)
If you want a good book on architecture, try The Architecture of Open Source Applications which you can read for free online. You could also watch Rich Hickey talk about how he designed Clojure: Hammock Driven Development for inspiration.