I've been using box2D for a few months, but I now badly need a physics library which can use boolean shapes (ie; cutting a circle out of a square) and/or curved surfaces. Does anyone have any suggestions?
Edit: Language is c++
Have you considered combining standard box2d shapes/fixtures and convex polygon attaching them to same body?
If you want you can even implement concave decomposition algorithms like the ones provided by this .NET(C#) Box2d port (at the bottom, it also supports some boolean operations):
PS: there should be some performance/complexity reason(s) for not supporting concave polygons as is.
I'm not sure there's such a thing as what you're describing. Everything is/can be made out of convex shapes, especially in graphics/physics math (this makes operating on bodies much easier)
It's more about finding, making, and implementing algorithms that break complex shapes down into as few simple shapes as possible. This is called Polygon Partitioning, and to do this i would suggest this simple, nearly includable library:
Physics math can much more accurately work with curved items such as circles though, as there are very quick equations when shapes have a common radius. I would suggest simply putting circles (or only a section of a cirlce) and polygons together to form whatever curved, pointy, or other crazy shapes you want to make. And again there are most likely algorithms for this out there even now.
you can also put "holes" into shapes by adding a "hole" shape to a body, and then test at run time whether a colliding shape is inside the whole or not. Then don't do any collision if it is.
also if you didn't know; sine and cos waves are very helpful for making curved hills and such.
The term you are looking for is
If you can live without curves (at least in your cut-out areas), go for the approach @Den suggests.
If you need to be able to slice polygons along curves, you are pretty much going to have to find a CSG library such as GPC. And you are going to have to be able to find one that supports splines in its intersections. And then you are going to have to transport that spline data into Box2D. And be warned -- good CPG libraries cost several hundred to several thousand dollars. There are free ones out there but they are by no means as full featured or as well optimised (in general, though they are not intended for real time use).
One thing I will warn you on -- if you don't have at minimum a graduate level background in mathematics, do not attempt a custom solution to the polygon intersection problem at home. There is a reason they charge big bucks for these solutions.