You're mistaking restitution. Or more specifically, it depends upon Bullet's chosen restitution implementation.
Ideally you would be able to define [ball, plane].restitution as 0.9 and [box, plane].restitution as 0.1, for example.
The total coefficient of restitution, in real life, is almost always >0.0 and <1.0.
Thus, when generating your restitution from two separate object materials, depending on the method*, ensure that your coefficients are between 0.0 and 0.5.
*The example above assumes that the two materials' restitutions are added together. I am not sure about whether Bullet adds them together, multiplies them or performs some form of min/max.
Further investigation:
- Looking at bullet's constraint solver you can see that it gets the contact manifold
- Once it has the manifold it gets a specific contact point
- Once it has the contact point it requests its "m_combinedRestitution"
However, I do not have the time right now to download the source and investigate where the m_combinedRestitution is set. If you were to investigate, I'd look into the NarrowPhaseCollision section of bulletNarrowPhaseCollision section of Bullet.