Outbound traffic Severe Slowness/Packet loss via S2S Tunnel

Question

Outbound traffic Severe Slowness/Packet loss via S2S Tunnel

Medicus 0

We are seeing severe packet loss from outbound traffic issues across the site to site tunnels in our Azure environment. All traffic from our office to Azure is fine, with 0 to minimal packet loss. Azure to our office is drastically slower, with at times 50% packet loss. This causes major issues copying/opening files on our File Server in Azure. We are lucky to get 1MB/s copying from that Azure server to our office, but copying to the Azure server is normal for a 1 gig link, at 18MB/s.

This is causing and has caused lots of frustration and slowdowns for our office.

We found this information out by running iperf3 commands between the Azure server and multiple computers at the office.

We replaced our firewall, and network switches, and rebooted our ISP's equipment, got a case open with the network equipment provider and they are all seeing everything normal. An inbound 800Mb/s transfer to the server, but at MOST 56Mb/s outbound from the server is insane

We also did a copy between 2 servers in Azure on the same LAN, and that was normal as well. This lead us to seeing that outbound traffic issue from the Azure servers.

Is there a config issue with the VPNGW, VPNIP, the Connection resource, or the Local Network Gateway?

0 comments

2 answers

Your answer

Answer 1

Quick unit check first, since it changes how bad the numbers look: 800 Mb/s inbound is about 100 MB/s, 56 Mb/s outbound is about 7 MB/s, a 1 MB/s file copy is only about 8 Mb/s, and 18 MB/s is about 144 Mb/s, roughly 15% of a 1 Gbps link (Mb/s divided by 8 = MB/s). A one-direction collapse with packet loss on an IPsec tunnel is very often MTU/MSS, so I'd test in this order. First, find the real path MTU from the Azure VM to an office host with ping -f -l 1400 <office-ip> and lower the size until it stops saying "Packet needs to be fragmented"; if it fails well below 1400, that's your sign. Second, on the on-prem VPN device clamp TCP MSS to 1350 on the tunnel interface (the value Microsoft recommends for Azure VPN gateways), then re-run iperf3 -c <azure-ip> -R -P 8 -t 30, where -R makes the Azure side send so you test the bad direction from the same client. Third, if it's still slow, check the gateway SKU: Basic tops out around 100 Mbps aggregate and per-tunnel throughput is lower than the headline number, and the Gateway S2S Bandwidth and Tunnel Egress Packet Drop Count metrics will show whether you're hitting a cap or real drops. If MSS clamping fixes iperf but SMB copies are still slow, try robocopy /MT:8, since a single SMB stream over a high-latency tunnel rarely fills the pipe.

Answer 2

The next action is to correlate a bidirectional iPerf test with the VPN Gateway’s tunnel egress-drop and bandwidth metrics, which will distinguish Azure-side tunnel drops or saturation from loss on the Internet/on-premises return path.

The tests already show that the VM-to-VM Azure path is healthy and that the problem is direction-specific, so replacing or retesting the office switches and firewall is unlikely to add useful evidence.

1. Measure the affected tunnel during a controlled test

In Azure portal → VPN gateway → Monitoring → Metrics, run an Azure-to-office iPerf test while graphing these metrics, split by ConnectionName, RemoteIP, and Instance:

Tunnel Egress Packet Drop Count
Tunnel Egress TS Mismatch Packet Drop
Tunnel Egress Packets
Tunnel Egress Bytes
Tunnel Bandwidth
Gateway S2S Bandwidth

Then reverse the iPerf direction and compare the same interval. Azure exposes separate ingress/egress drop counters and a specific traffic-selector mismatch counter, so this is more conclusive than throughput alone.^1^

Microsoft’s documented iPerf method uses 32 parallel TCP connections and then reverses the client/server roles:

:: Receiver
iperf3.exe -s -p 5001

:: Sender: run from each side in turn
iperf3.exe -c <receiver-private-IP> -t 30 -p 5001 -P 32

Run it during a nonpeak period and save both results. Parallel streams prevent a single TCP stream’s window/latency behavior from being mistaken for tunnel capacity.^2^

Interpret the results as follows:

Egress drop counter rises: the gateway is recording outbound tunnel loss. Proceed to packet capture and diagnostics.
TS mismatch rises: compare the Azure connection’s traffic selectors/address prefixes with the on-premises encryption domains.
Bandwidth reaches the SKU/per-tunnel range without drops: capacity or cryptographic performance is the likely constraint.
Gateway emits packets with no Azure drop increase, but the office does not receive them: investigate the public Internet path or on-premises VPN endpoint rather than the Azure LAN.

2. Compare the result with the actual gateway SKU

Check VPN gateway → Overview → SKU. Gateway throughput is aggregate across S2S, VNet-to-VNet, and P2S connections; one tunnel can achieve substantially less than the aggregate headline benchmark. Performance also varies by IPsec algorithm—for example, published per-tunnel results are higher with GCMAES256 than with AES256/SHA256 and much lower with DES3/SHA256.^3^

Calculate the ceiling as the lowest of:

VM network throughput
VPN Gateway aggregate/per-tunnel performance
On-premises VPN device encrypted throughput
ISP/path capacity

Do not upgrade the gateway solely because the office circuit is 1 Gbps; first confirm that the metrics show saturation rather than packet loss.

3. Test for MTU/fragmentation

The one-direction collapse is consistent with—but does not prove—a Path MTU Discovery issue. IPsec adds headers and reduces usable MTU. If ICMP “Fragmentation Needed” is blocked, oversized packets are repeatedly dropped and retransmitted.^4^

During a test window:

Configure TCP MSS clamping to 1,350 bytes on the on-premises VPN device.
If MSS clamping is unavailable, test a tunnel-interface MTU of 1,400 bytes.
Re-run the same bidirectional iPerf test.
If performance immediately normalizes, retain the supported setting and confirm that required ICMP messages are not being blocked.

Apply this on the VPN device/tunnel, not by indiscriminately increasing the Azure VM MTU.

4. Check asymmetric routing only if the gateway is active-active

If VPN gateway → Configuration shows active-active enabled, Azure can send a flow through either tunnel and does not guarantee that return traffic uses the same tunnel. Azure handles this, but an on-premises firewall that requires strict symmetry might drop or mishandle the flow.^5^

Split the metrics by Instance and RemoteIP. If loss occurs only through one gateway instance/tunnel:

Compare routes and traffic selectors for both tunnels.
If using BGP, adjust route preference so both directions select a compatible path.
As a controlled test, use active-standby only if the current availability design permits it.

This branch does not apply to an active-standby gateway.

5. Capture evidence and escalate

Enable diagnostic settings for:

GatewayDiagnosticLog
TunnelDiagnosticLog
RouteDiagnosticLog
IKEDiagnosticLog

Send them to Log Analytics or Storage, then reproduce the issue.^6^

Also run a filtered VPN Gateway packet capture for the Azure server and office client. Gateway capture supports inner/outer traffic, one-way or bidirectional capture, and five-tuple filtering.^7^

Capture simultaneously at:

Azure VM
Azure VPN Gateway connection
On-premises VPN device

If Azure’s gateway capture shows packets leaving without gateway drops but they never reach the on-premises WAN capture, provide Microsoft and the ISP with synchronized captures, timestamps, connection name, remote IP, gateway instance, and metric exports. If the gateway capture itself shows unexplained egress loss below documented capacity, open an Azure support case with that same evidence.

References