First of all - let me say - I have done exactly this (developing on PC and then porting to Xbox 360 late in development) and had only minimal trouble.
With XNA 4.0 and the introduction of the profiles (HiDef is what you want to target for the 360) it has become significantly easier to keep within the specifications of the Xbox.
You basically need to remember that the Xbox 360 has different performance characteristics to a PC. So while your game may port fine, you may have to spend some time doing performance tuning, especially if you are doing anything "fancy".
By far the biggest performance gotcha is the garbage collector on the Xbox 360. It is slow - certainly a lot slower than the one on the PC. So when you are developing on the PC you will want to use the Microsoft CLR Profiler to profile your memory: Once your game has loaded and is running you want to see close to zero allocations (XNA on Windows will still allocate a few things, usually to do with input, that are not a problem on the 360). Here is a blog post about avoiding garbage collector issues.
I recommend buying some Xbox 360 controllers early on, and using those during PC development.
The other thing to be aware of when developing for the 360 is the The Evil Checklist that your game is checked against during peer-review.
You will notice that there's a lot of stuff in there to do with profiles, the guide, the marketplace and particularly storage devices. These cause issues that just don't come up on PC. If your game stores data, does networking, needs to interact with profiles, etc, I would recommend getting an Xbox 360 early in your development cycle. Also - remember to build in support for a demo, as it is required for XBLIG.
I also can't speak much about Torque-X, but I assume that it's just a bunch of libraries that use XNA - so it should not cause any issues.
