Not sure which aspects of Matrix math you're having difficulties with, so I'll list two books that helped me get over the 3D math hump. Chances are, you'll need to work through more than one book to get everything to finally "click".
- 3D Math Primer for Graphics and Game Development This is a great book, everything is explained thoroughly and in a language that's easy to understand. I would start here.
- Mathematics for 3D Game Programming and Computer Graphics Another good 3D math textbook. Covers a wide range of material. It's concise and straight to the point.
I would start with these, even if you have to re-read the material a couple of times. I spent months going over 3D math topics before being comfortable with it.
EDIT: I posted this before you updated your question. If you're looking for help directly with how to use Matrices in XNA, check out XNA Game Studio 4.0 Programming: Developing for Windows Phone 7 and Xbox 360 by Tom Miller & Dean Johnson (can't link, due to spam prevention, but it's on Amazon). Chapter 4 goes into the very basics of 3D Math and directly relate to the math classes in XNA. You'll probably learn a lot from the code samples in the later chapters as well.
EDIT 2: Another book that I recommend as well is Microsoft XNA Game Studio Creator's Guide. While this only covers XNA 3.0, the majority of it is still applicable to XNA 4.0. It not only has a chapter on Matrix math, but on creating a graphics engine camera system that shows great use of Matrices in XNA. I think you'd take away a lot from that sample.