This is why game servers always use UDP instead of TCP for time sensitive data.
TCP is used for initial connections, game lobbies, and parameters, shared variables... Once TCP crosses a multi-homed router, timing becomes unimportant.
The negotiation, reliability, sequencing are all the trademarks of TCP. TCP requires acknowledgements for every packet(within a window), and anything received out of order will wait until it is in order.
Use UDP and ignore out of order sequence packet numbers. The usual solution is to include the last couple of frames deltas from and with the current frames absolute data.
Many papers have been written on the topic, that includes anti-cheating.
Solve the problem by adding timestamps, and ignoring or blending late frames.
The accumulation problem cannot be solved without a separate connection or 2 or 3...