What you're seeing is the result of lazy state changes and your swapbuffers call flushing the pipeline.
In the first place, most drivers won't actually change state when you make a GL call; instead they just record the fact that you requested a state change (and may set a dirty bit to indicate that the state actually needs to be changed). Then, when a draw call happens, all of the recorded state changes are applied. The result of this is that your timings for any glDraw* call may widely vary, some taking no time at all, some seeming to take a lot of time. If you're using VBOs it may be even more misleading as the timing may bear no relation whatsoever to the amount of geometry being drawn.
So if you've a lot of state changes queued up for that first draw call in a frame, then the glDrawElements may seem to take longer.
Secondly, any swapbuffers call will flush the pipeline, meaning that any queued up work will start to be processed. Because your glDrawElements call comes next, it may be waiting on that queued-up work to finish before it can proceed, so it would give aberrant timings if so.
You can test for and remove this factor from your timings by putting a glFinish call after your swapbuffers. This will (assuming your driver is conformant) complete all the outstanding work before it returns, so that it will no longer pollute your timings. If you have a bunch of CPU-side work you can do before the next frame runs, here is a good place to do it, as it can give the pipeline more time to complete it's flush before you start drawing stuff again.
In general however what you've got here is a classic case of CPU timings not being a good indication of what's actually happening on the GPU. Because a CPU and GPU operate asynchronously, all that you're measuring is the CPU-side times; you have basically got no valid data whatsoever for what's happening on the GPU.
