It seems you want new types of Materials to be able to be created at runtime. Then I don't see the problem with having a pointer to the Material of the object; it's how I would do it. Perhaps elsewhere in the program (e.g. in the factory class which creates these Object instances) you would have an array of all the Materials, to keep track of them and allow for choosing between them. Also you would need to either never delete a Material once it's created, or keep track of the references to a material so that if you want to delete the material then you reassign new materials onto all the objects which were previously using that material. You certainly don't want to
delete a Material in memory, only to have Objects still pointing to that memory location which is now invalid!
The problem with the integer ID which is a position of an array, for example, is if that array is manipulated, for example if a material is deleted, then you need to either put a 'null' in that position and never use it again, or you need to loop among the objects and change their material ID when you change the array. So for example if you have three materials, and an object points to material 1, then if you delete material 1 you need to either put a null in that place, or change the object id to e.g. -1 before you allocate a new material in position 1 (or else the object is magically given a new material!).
Also an std::map seems like unnecessary overhead in this case. It seems to me that you don't really need the map when you could use a pointer and reference counting as mentioned above. You could certainly use it though, keeping in mind similar downfalls such as mentioned in the previous paragraph; e.g. you still need to make sure the IDs are unique among all materials deleted or not, or else you could get objects whose materials get reassigned just because the material gets deleted and a new one gets put in with the same ID as the old one.
If however Materials can't change at runtime, you could of course make an enum with the material type, and the object would have an element of that enum (or a pointer to that element; my c++ is rusty).