intermittent connection failures under high concurrency

Use this troubleshooting path to determine whether the failures are caused by outbound port exhaustion, connection limits, or application connection behavior:

1. Check whether the symptom matches outbound connection exhaustion. Common signs include intermittent time-outs, slow responses, intermittent 5xx or Bad Gateway errors, and inability to connect to external endpoints. These are documented symptoms when outbound connection limits or SNAT ports are exhausted in Azure App Service.
2. Determine whether the issue is on Windows outbound ports. On current supported Windows versions, the default dynamic client port range is 49152 through 65535, which provides about 16,384 ephemeral ports for outbound connections. Windows normally holds closed ports for four minutes before reusing them. Under high connection churn, this can cause intermittent connection timeouts.
3. Identify the worst offenders by collecting port ownership. Use the ports.txt approach described in the SQL Server connectivity troubleshooting guidance to list inbound and outbound ports, port numbers, process IDs, and owning applications. Review that list to find which process is consuming the most outbound ports and whether the host is approaching the port limit.
4. Check whether the application is rapidly opening new outbound connections instead of reusing them. The guidance explicitly states that exhausting outbound ports is often caused by non-pooled connections or non-pooled background connections. The preferred fix is to change the application to use connection pooling.
5. If the workload is hosted on Azure App Service, check for SNAT exhaustion. Each App Service instance is initially given 128 SNAT ports. This limit especially affects repeated outbound calls to the same destination address and port. Closed connections are reclaimed only after four minutes. If the app rapidly opens new connections to the same endpoint, intermittent outbound connectivity issues can occur.
6. If immediate application changes are not possible, adjust Windows port reuse settings carefully. As a temporary mitigation on the host machine, TcpTimedWaitDelay can be reduced from 240 seconds to as low as 30 seconds so outbound ports are recycled faster. The dynamic client port range can also be widened by using netsh int ipv4 set dynamicport tcp or the IPv6 equivalent. Risk: these are system-level networking changes and should be tested carefully because they affect host networking behavior. They do not fix inefficient connection usage in the application.
7. Consider whether antivirus or network filter drivers are interfering. If packet traces show packets arriving but no ACK being sent back, the guidance indicates antivirus or another network filter driver may be dropping or delaying packets long enough to cause failures.
8. For application design, reduce unnecessary simultaneous connections. Windows Winsock guidance recommends that concurrent connections generally should not exceed two except for special-purpose applications, with a rule of up to four short-lived connections or two persistent connections per destination. Reusing persistent connections is more efficient than creating many short-lived ones.
9. If testing on IIS running on a Windows client OS, verify platform request restrictions. IIS on Windows client OS has request concurrency limits and can quickly exceed them, causing slowdowns and in some cases HTTP 403.9 or queued requests. Windows Server OS does not have these IIS request restrictions.

A practical decision tree:

• If failures happen during outbound bursts and recover after a few minutes, suspect ephemeral port exhaustion or App Service SNAT exhaustion.
• If one process owns most outbound ports, focus on that application's connection pooling and reuse.
• If traces show missing ACKs, inspect antivirus or network filter drivers.
• If the issue is specific to IIS on Windows client OS, verify request concurrency limits.

References:

• Troubleshoot intermittent outbound connection errors in Azure App Service
• Troubleshoot intermittent or periodic issues with connecting to SQL Server
• TCP/IP-specific Issues
• IIS Request Restrictions on Windows Client OS