I saw there are quite a number of articles and sample code regarding ball bouncing within a rectangle.
Is there any articles or sample code regarding ball bouncing within a polygon? I try to look into Box2D demo, I can't find one yet.
I don't know of any articles, but hopefully you can figure out how to rig it up from these suggestions below:
If it is a convex polygon, a convex hull is about as simple as it gets while still being a general solution. That topic is extremely well documented and I think every physics library has a "plane" primitive that can be used to build it. Basically, you have N planes for an N sided figure. Each plane divides space in half, the part in 'front' of the plane, and the part 'behind' it. If you had two planes making an 'X', the area that is in front of both forms a 'V'. If you add a third plane such that the front faces the inner point of the 'V', you get a closed off area that is ' in front' of all three planes. This is a triangle, and the planes form its convex hull. This extends to N-sided convex polygons fairly easily.
If it is a concave polygon, then things are a bit hairier but still manageable: just use an array of line segments. You can simulate that if a line segment isn't available by having very "thin" rectangles. This would also work for convex polygons, but it isn't as clean or as fast.
I hope this helps.