Draw calls have a fair amount of CPU overhead, as the driver has to do a bunch of work internally on each call, so reducing the number of draw calls is often important for CPU performance. For the GPU, the number of vertices, triangles, textures etc. matters more than the number of draw calls. Conversely, the size of the draw calls doesn't matter very much for the CPU, only how many of them there are.
The rule of thumb that I've seen quoted often for DX9 (which XNA runs on) is no more than about 2,000-3,000 draws per frame to avoid becoming CPU-limited. YMMV, of course, but the limit is going to be on the order of several thousand, definitely not a million.
With very careful renderer design it's possible for games to push more draws. Firaxis gave a GDC talk describing how they achieved 15K draws in DX11. Another question here noted that Saints Row 3 apparently gets 7K draws in DX9.
In DX10-11 the CPU overhead of draw calls is supposed to be less than in DX9, but I haven't seen a measurement of how much less. The sense I have from talking to people about it is that this doesn't make a big difference, i.e. the draw call limit is still only a few thousand. DX11 also allows multithreaded draw call submission (deferred contexts), which is supposed to let you distribute the CPU overhead over multiple cores. However, internally the graphics driver may still be processing things with a single thread, so DX11 deferred contexts may not be very effective either.
On consoles, due to their lower-level APIs (which avoid a lot of layers of validation and conversion in D3D), you can use multithreading much more effectively and reach 10,000-20,000 draws per frame without being CPU-limited. (All of that doesn't help you for XNA; I mention it only for completeness.)