Rigid body physics is actually quite simple in concept - that is, the behaviors an engine is trying to create are not very complicated. That's probably why the libraries you looked at seem quite simple to you.
The trick is that making a sim that is robust, stable, and fast, even in the presence of things like floating-point error and low, variable framerates, is quite difficult and usually involves a great deal of tweaking and fiddling. Further complicating the issue is the fact that a lot of extremely smart people have made heroic efforts at finding better solutions, and have come up with beautiful and powerful but difficult-to-understand algorithms that have given the field a reputation for black magic.
In general, a physics engine is going to be concerned with three things (the order in which they do them in the internal loop varies):
- Integration (approximating the Newtonian equations of motion),
- Collision detection, and
- Constraint solving (updating positions and velocities to satisfy nonpenetration or other user-specified constraints.)
For most of these, there are both simple but not-so-accurate algorithms and complex but more-accurate algorithms. To get you started:
- For integration, most people use either Symplectic Euler or Verlet (which are actually equivalent. You can read up on numerical integration if you like, but the consensus seems to be that cheap first-order methods are fine.
- You can find all the collision detection references you could want on the Real-Time Rendering resource page.
- For constraint solving, the method of sequential impulses is very easy to understand. Erin Catto (the author of Box2D) gives a presentation at GDC every year - the slides are linked from the Box2D main page. The 2006 slides cover sequential impulses pretty well. You can also look into fancier methods like Featherstone or projected Gauss-Seidel.
There's an excellent introduction to simple methods and what can go wrong in the early chapters of the just-published book Game Physics Pearls.