The main thing to remember when dealing with an RTS game is that collision detection isn’t the same kind of animal it is in a FPS. The game, not the player, determines if a unit actually hits its target, so you’re not worrying about things like "If a shot goes between his legs, how do we know it didn’t hit him?" but rather "we decided that he hit him, so make the shot look like it hit him." Units attacking one another usually amounts to "Is the unit in range? Does it have line of sight? Okay, animate the attack and calculate damage." Most RTS games don’t even taken into account a "chance to hit" all attacks hit, and damage is random within some range which expresses the idea of "how on target" the shot was. This means that for combat you’re usually just considering ray-polygon intersections to check that the units have line of sight. This can be performed against a very simplified version of the map geometry using bounding boxes (A single box for each unit if you want them to be included, rough outline of the terrain and map elements). There are a huge number of tutorials on this, just search for Ray-Box Intersection, Ray-Polygon Intersection and Bounding Box Collision Detection to find out more.
Movement in RTS is generally the more complicated bit, for the same reason combat is simpler. The game, not the player, has to navigate the terrain. Generally this is accomplished by creating a kind of "map of the map" which defines areas as passable, impassable, (and possibly shoot-through-able or not which can be used to further simplify line of site determinations). This map is then used to plot paths for the units. Google Navigation Mesh for details on how the “map of the map” is constructed. Google Pathfinding for more details on how to calculate and plot paths. Dijkstra's Algorithm and A* are two common pathfinding algorithms. Pathfinding is a part of the mathematics field known as Graph Theory if you’re interesting in reading up on the topic more broadly.
As for physics, except in AAA titles there usually isn’t much real physics implemented beyond basic velocity/acceleration calculations for movement. You can store values in the passable areas of the map to indicate high and slow the unit’s velocity when they are moving from low to high terrain, or through special terrain. When units are damaged they are playing a hard-coded animation of “being flung through the air” or “tossed aside” etc, based on the outcomes of the combat calculations.
If you have any specific questions just let me know.
