If you understand the general programming fundamentals work - datatypes, classes, functions, etc - I would say go for it.
When I started learning C#, I did have games in mind, but started with developing console apps that, while they looked very basic, incorporated some advanced C# techniques, in order to better my understanding of the language.
Once you have an understanding of how C# works, I'd go ahead and download the XNA studio from here - http://www.microsoft.com/download/en/details.aspx?id=23714
You will want to start with learning SpriteBatch. Some good tutorials can be found here...
From there you can start experimenting with pixel shaders (on SpriteBatch, http://msdn.microsoft.com/en-us/library/bb313868(v=xnagamestudio.31).aspx ), and also matrix transformations ( http://www.david-amador.com/2009/10/xna-camera-2d-with-zoom-and-rotation/ ).
Once you have a good grasp of that, you can finally move on to using the graphics device to draw primitives and models using BasicEffects and custom shaders.