If I add a math library (for example containing a Matrix class) and use it in my program drawing with OpenGL, will my be work slower than if I used standard OpenGL functions for matrix calculations? Does the same hold true for DirectX?
Matrix manipulations are always done on the CPU. It doesn't really matter though, the total amount of floating point operations you'll do involving matrices will be less than 1000 and any modern CPU will deal with that in a few nanoseconds. Using an external library which is probably optimized is absolutely worth it, if only for the readability improvements. Not to mention that matrix manipulations in OpenGL are deprecated so you're obligated to write your own.
I can't give you any real data, but I profiled a program I wrote some time ago. It used a matrix library I wrote myself, without any optimizations. The matrix calculations didn't even show up in the profile.
So, don't worry about it unless you're absolutely sure it's a problem.
To answer the first question: on both
Usually not at all much on the CPU and then at least once per vertex (usually more) on the GPU.
I'd not care about speed except if you do heavy work with matrices in the (c++) code. If you don't (you just calculate worldView projection and so on, say some handfuls for each model at most) just use what you feel best at ease with.
The day you start to use a Lot of matrix multiplications, that day you'd know yourself what you need to do to speed up code (and if you need to speed up the "code" or the shader or maybe other things like profiling bandwith). If that day you don't know, then you are probably doing something wrong.
What about DirectX?
Well, if you want to be compatible with both then go for a library (I use Irrlicht which is a whole graphic engine) so you can switch easily (beware though, OpenGL is righthanded and DirectX is lefthanded, shaders usually don't accept a simple switch between those if you use "complex" calculations like normal maps).