Remember that FPS is not a great metric for performance: it's not a linear measure. 60 FPS is ~0.016 seconds per frame, whereas 40 FPS is 0.025 seconds per frame. That's a different of only 0.009 seconds for fourteen times as much render throughput. Timing rendering (which executes on the GPU) is not as straightforward as timing CPU operations. See this guide, which covers D3D specifically, but explores some concept that will apply to GL as well.
60 FPS for almost no objects sounds like you have v-sync enabled. In that scenario, you don't need too much more stuff to be rendered before you can hit a threshold where you have to block for a while waiting for the (modern equivalent of the) vertical blank interrupt before you can continue. This can make your FPS appear to drop dramatically. If you disable v-sync you may notice a much less severe drop.
All that being said, all the GPU will do for you is clip triangles outside of the viewport. It will not do frustum culling for you, which means every bit of geometry you submit will make it all the way through the vertex transformation pipeline into clip space at the least. The fragment shader should not run on primitives that have been clipped out.
If you implement frustum culling yourself (the API will not do it for you), you can boost your performance in general by not having to even submit out-of-view geometry to the card.