I know of three that still seem reasonably actively developed: Spring, OpenRA and Stratagus. Wikipedia has a list of open source game engines, some of which may be appropriate for an RTS game (such as FIFE, which used to bill itself as a Fallout-style engine, but should still be appropriate for RTS games).
Some of these engines may work directly with 3DS Max model files, but that file format (.3ds) is actually pretty archaic, cumbersome, and suboptimal for direct consumption by a game engine, so you may find that the engines instead support some other model format that Max can export to (or supply their own conversion tools).
Also, it's not open source (
and does in fact cost money might cost if you need certain features), but people have used Unity to make RTS games as well.