Troubleshooting – MTU Path Discovery

Every network interface has a maximum packet size called the interface’s MTU (Maximum Transmission Unit). The full path from one computer to another may travel across many links with different MTUs. The smallest MTU for all the links in a path is the path MTU.
If a packet starts out on a network device with a large MTU, it may arrive at a device interface with a smaller MTU and be too big to fit.

If during SSL Check you see that request is stuck and the certificate details are not returned or Inspite of telnet connectivity test passing you see that cURL request is stuck a potential isue could be MTU. Take the following steps to verify and resolve MTU related issues.

Sender ----> Network Link ---> Network Link ---> Network Link ----> Destination

when a device interface in between has a lower MTU then it sends an ICMP (type 3 code 4 ) message back to the sender (in this case the client) stating that Fragmentation is needed. IF ICMP is disabled in customer’s network then these ICMP notification will not be seen at the sender.

======= ICMP Enabled =======

The client side packet captures should show the fragmentation needed ICMP packets from the device in between and the IP header would give the IP of the this hop in between in the source IP field.

If ICMP is enabled, and you want to know the appropriate MTU size to configured at the sender can use extended pings in which with different tests packet size can be increased and find the maximum packet size they can go up to .

With extended ping, send the largest packet you can, and if it won’t fit through some link get back a notifications. The notifications arrive as ICMP (Internet Control Message Protocol) packets known as “fragmentation needed” ICMPs (ICMP type 3, subtype 4). The notifications are requested by setting the “do not fragment” (DF) bit in packets that are sent out.

####### Windows ########

ping IP_ADDRESS -f -l 1800
Pinging IP_ADDRESS with 1800 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for IP_ADDRESS:
Packets: Sent - 4, Received - 0, Lost - 4 (100% loss)


ping IP_ADDRESS -f -l 1400
Pinging IP_ADDRESS with 1400 bytes of data:
Reply from IP_ADDRESS: bytes=1400 tine=39ms ITL=54
Reply from IP_ADDRESS: bytes=1400 tine=26ms ITL=54
Reply from IP_ADDRESS: bytes=1400 tine=39ms TTL-54
Reply from IP_ADDRESS: bytes=1400 tine=40ms TTL-54
Ping statistics for IP_ADDRESS:
Packets: Sent - 4, Received - 4, Lost - 0 <0% loss>)
Approximate round trip tines in milli-seconds:
Minimum - 26ms, Maximum - 40ms, Average - 36ms

The above test gives an idea that MTU is between 1400 and 1500. We need to continue to find the correct MTU.

####### Linux #######

ping -M do -s 2000 example.com
PING example.com (93.184.216.34)2000(2028) bytes of data. ==> In addition to 2000 byte packet 28 bytes are added of header
ping: local error: Message too long, mtu=1500 ===> Shows the MTU on the sender
ping: local error: Message too long, mtu=1500
--- example.com ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1000ms


ping -M do -s 1500 example.com ===> FAILS
PING example.com (93.184.216.34) 1500(1528) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
--- example.com ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2052ms

ping -M do -s 1472 example.com ===> Passes (1472 Byte Packet + 28 ByteHeader = 1500)
PING example.com (93.184.216.34)1472(1500) bytes of data.
1480 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=1 ttl=50 time=1.33 ms
1480 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=2 ttl=50 time=1.40 ms

--- example.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms

# MAC

ping -D -s 1500 example.com
PING example.com (93.184.216.34): 1500 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
Request timeout for icmp_seq 0
--- example.com ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss


ping -D -s 1200 example.com
PING example.com (93.184.216.34): 1200 data bytes
1208 bytes from 93.184.216.34: icmp_seq=0 ttl=42 time=245.920 ms
1208 bytes from 93.184.216.34: icmp_seq=1 ttl=42 time=206.379 ms
--- example.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss

ping -D -s 1223 example.com
PING example.com (93.184.216.34): 1223 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
Request timeout for icmp_seq 0
--- example.com ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

ping -D -s 1222 example.com
PING example.com (93.184.216.34): 1222 data bytes
1230 bytes from 93.184.216.34: icmp_seq=0 ttl=42 time=205.911 ms
1230 bytes from 93.184.216.34: icmp_seq=1 ttl=42 time=221.415 ms
1230 bytes from 93.184.216.34: icmp_seq=2 ttl=42 time=207.312 ms
--- example.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss

The above test I ran on my MAC OS gives an idea that MTU is between 1500 and 1200. Recursively I was able to find the correct MTu as 1222.

======= ICMP Disabled =======

1) If ICMP can be enabled on the network. Enable ICMP on network and run the extended ping test.

2) If ICMP cannot be enabled on network , Try traceroute –mtu or tracepath -n
to see if a hop in between has lower MTU than what was deduced using the packet capture for client and server.

tracepath -n Host
1: x.x.x.x 0.097ms pmtu 1500
1: x.x.x.x 0.535ms
1: x.x.x.x 0.355ms
2: x.x.x.x 0.430ms pmtu 1400
2: x.x.x.x 0.763ms reached
Resume: pmtu 1400 hops 2 back 254

In above sample an intermediate network link has lower MTU (1400) than the sender(1500).

If an intermediate device has a lower MTU then the sender then reduce sender MTU and try again.