You can build XNA projects outside of the IDE (debugging them effectively is another matter entirely), just like any other C# project. Using the MSBuild system.
The specific thing that you need to know about for XNA is the Content Pipeline that you need to process your content. You can build content projects outside of the IDE, but to set them up you would have to edit the
.contentproj file by hand (it's XML). The values for that file are exposed nicely through the IDE (using reflection). Figuring out what to set them to yourself will be an exercise in frustration.
Also, I don't think XNA Game Studio will install without Visual Studio also being installed. I don't think there's any other "within the EULA" method of getting the DLLs necessary to run the content pipeline.
So, in the interest of sanity and practicality, you should probably at least install Visual Studio, even if you insist on not using it. Although, once you've got it installed - you should give it a proper go - it's rather excellent. For XNA you probably want VC# 2010 Express.
I haven't checked on MonoGame for a while, but last time I looked you needed to use the XNA Content Pipeline to build a few kinds of content, like fonts. I have heard that they're working on their own content pipeline solution - not sure how far along that is coming. But otherwise the situation should be similar to XNA.