Why do TCP loopback sends sometimes stall for ~10 ms with TCP_NODELAY enabled?

Tadas Pranckevicius 0 Reputation points
2026-11-03T20:01:47.4966667+00:00

We have a small C++ reproducer where TCP sends over 127.0.0.1 occasionally stall for around 10 ms. It uses two processes and plain Winsock, with no third-party libraries.

This happens with both blocking send/recv and overlapped WSASend/WSARecv using IOCP. TCP_NODELAY is enabled at both ends.

The test runs on Windows 11 Home 25H2, build 26200.9711, on an Intel i9-12900HK. It sends 32-byte messages in both directions over one TCP connection: about 1,600 messages/s one way and 3,200/s the other. Each process has two sender threads and one receiver/completion thread. Socket buffers are 64 KB.

ETW shows the delay between TcpSendPosted and the first TcpSendTransmitted event for the same bytes. For example, one IOCP send looked like this:

  • WSASend returned after 8.6 microseconds.
  • The send completion was dequeued 16.7 microseconds after the call started.
  • TCP took 9.562 milliseconds from posting the send to transmitting it.
  • The peer received it 9.746 milliseconds after the call started.

We know a send completion does not mean delivery. What we are trying to understand is why the data sits in TCP for that long. We matched the TCP events by connection and byte sequence range, and the traces reported no lost events.

Using two connections, one for each direction of application data, changed the result. In a separate pair of 90-second tests, with the same planned workload and 428,276 messages per run:

  • One connection: 45 TCP send delays of at least 5 ms; worst was 12.943 ms.
  • Two connections: no TCP send delay reached 1 ms; worst was 0.789 ms.

Larger buffers and requesting an ACK for every packet did not eliminate the stalls. An earlier packet capture showed no zero/full receive window, and the slow forward messages did not need retransmission.

Some earlier traces show a send being posted while another CPU is handling received data on the same connection. That may be a clue, but we have not established the cause.

Is this a known Windows TCP behaviour? Is there a way to avoid it while keeping one bidirectional connection? If more tracing is needed, which events or stacks should we capture?

Source, build instructions and a runner are attached. Each test takes 90 seconds. We also have the original ETL captures and per-message logs available. [[[[run.py.txt](/api/attachments/38c2516f-cb4a-46d3-ba7c-6545654b6396?platform=QnA)](/api/attachments/88204164-fd54-4cd9-b353-959184adfb1c?platform=QnA)](/api/attachments/7ae35d0f-2132-4925-83cb-218e1becd7be?platform=QnA)](/api/attachments/b6f53226-4c64-4197-aed7-ec472f8e2d34?platform=QnA)

Windows development | Windows API - Win32
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Zack Nguyen (WICLOUD CORPORATION) 405 Reputation points Microsoft External Staff Moderator
    2026-11-04T02:15:46.5366667+00:00

    Hi @Tadas Pranckevicius, 

    Here are my answers to your questions after reviewing on them along with the available Microsoft documents: 

    1. Is this a known Windows TCP behavior? 

    The public documentation doesn't describe this ~10 ms stall as expected behavior or as a known issue on your build. TCP_NODELAY disables Nagle's algorithm, but the documentation doesn't promise a maximum transmission delay (IPPROTO_TCP socket options). 

    The two-connection result suggests the delay is linked to send and receive activity happening on the same connection. It doesn't prove a specific cause, such as lock contention, a timer or scheduling. Given your ACK, window and retransmission observations, delayed ACKs alone also don't explain it. 

    2. Can it be avoided while keeping one bidirectional connection? 

    The current evidence doesn't confirm a single-connection fix. If both sender threads call send concurrently, it's worth testing with one sending thread per process. Keep the receive path and workload the same. 

    The WSASend documentation advises against calling WSASend from multiple threads at the same time on one stream socket, because some providers can interleave the data. That warning is about data ordering, so it doesn't show that concurrent sends cause the stalls. If your sends are already serialized, you can skip this test. 

    Splitting the traffic across two connections reduced the delays in your runs, but that hasn't been proven as a fix. Avoid SIO_LOOPBACK_FAST_PATH, because it's deprecated (SIO_LOOPBACK_FAST_PATH Control Code). 

    3. What additional tracing would help? 

    Capture these together on the same timeline: 

    • TCP events: TcpSendPosted, TcpSendTransmitted, and the receive/ACK events around them, with connection and sequence-range details. Use netsh trace show provider Microsoft-Windows-TCPIP to see which keywords are available (Using Netsh to manage traces). 
    • Winsock/AFD events: Add Microsoft-Windows-Winsock-AFD to match socket calls with TCP activity (Control of Winsock tracing). 
    • CPU and scheduling activity across the whole system: Capture CPU sampling, CSwitch, ReadyThread, DPC/ISR, and process/thread events. Collect stacks for CPU sampling, context switches and ready-thread events, plus TCP event stacks if available. Set these up in a custom WPR profile, then confirm the stacks appear in the ETL (Authoring Recording Profiles). 

    For one representative stall, look across all CPUs to find what blocks the send and what runs just before the data is transmitted. 

    To start, please share your current capture commands or WPR profile, one affected ETL and its matching message log. If those traces already include the events and stacks above, you may not need a new capture. Both runs' state.json files would also help confirm the one-connection vs. two-connection comparison. 

    I hope this helps answering your questions. If you find my response helpful, I would greatly appreciate it if you could follow the guide here so others experiencing similar behavior or issue can benefit from it as well.   

     

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.