Few years ago I tried to make a simple 3D game - billiards. Completed like 50%, stuck with physics. Basically, I only need to calculate balls rolling over flat surface, but it would be nice to make something more flexible. I know all the formulas and laws (most of them, anyway). the problem is I have no idea of how to make good physics engine architecture-wise. I tried google and other forums but didn't find what I was looking for. The only suggestion was to look at open-source engine, but I'm not that good a programmer to make heads or tails out of it...
How to start an Architecture/Design Task: With pen and paper.
Get yourself a large sheet of paper and start drawing out the components and items that will exist within your engine. What properties each entity will need so that you can model the physical interaction.
You'll need to figure out what entities your code will have to deal with, which entity has the responsibility of running the physics simulation (and which bit of the simulation).
Then you'll have to figure out how the entities are interacting. Use colours.
Now go to sleep. leave the design for a few days and do other stuff. If you have some friends you can talk to, this is a good time to ask them about what they think.
Return to you design. You'll see some obvious problems and will feel the urge to make it more elegant. Do it.
Start coding from your second design. Rinse and repeat until you feel proud of your design and implementation. Then ask for criticism from people whose technical skills you respect, and realise that you've barely scratched the surface and that your engine is held together with duct tape. Rinse and repeat.
There is no perfect formula for architectural systems like this. It's a matter of experience and a taste for elegance. Books may help you by exposing you to interesting ideas, but if you're comfortable with the physics, you're no more or less qualified than anyone else to design an engine.
Don't worry about a good engine worry about about getting an engine that runs, and move towards good from there. (even if it means you're hitting polynomial complexity )
Worry more about being able to communicate the way your engine works to someone else. Until you can explain your code to your mum, you don't understand what you're doing.
That last point is most important. When you get stuck, try to explain what your engine should do to a friend, on IRC, or in an email to someone whose work you respect. The act of trying to explain it to someone else will often reveal what your problem was in the first place. (You don't actually have to hit the "send" button, if you feel shy...)
If you want a good book on architecture, try The Architecture of Open Source Applications which you can read for free online. You could also watch Rich Hickey talk about how he designed Clojure: Hammock Driven Development for inspiration.