I'm working on a networked multiplayer game, initially for iOS.
Even with TCP_NODELAY there are large fluctuations in latency. I can't be sure of the reason, but I would not be surprised if it was resends on packet loss due to flaky connections.
Having done no real life server work with UDP as opposed to TCP (which I have lots of experience with), I wonder if there are any significant gains to trying a UDP-based architecture.
Client packets are typically only a few bytes, sent rarely, except when moving, when the player might issue 2-8 commands/s. Server packets are slightly larger and mostly sent as response to a client command.
Connections need to be reliable and ordered.
Very early on I did some exploratory tests with TCP versus UDP. I would run connections
Phone -> 3G -> External static IP of router -> Wifi -> Development computer and
Phone -> Wifi -> Router -> Wifi -> Development computer.
What I wanted to improve was mainly the very uneven lag one would experience moving around on the map with a non-local connection.
Obviously starting animations can help, but there are (rare but valid) reasons why a move command might be rejected that the client have no way to predict. In addition, much data is hidden from the client, which again gives very little room for predictive responses.
Anyway, my results was the UDP and TCP gave pretty much the same average latency when sending data.
What I didn't look at was the actual spread - for example the maximum TCP latency in face of packet loss. I would have have needed to do some packet recovery mechanism in UDP to figure out if UDP could do better.
So again the results I see today with TCP is that usually it works fine, then suddenly a slowdown and a burst of movement as all the other buffered packets are more or less executed at the same time.