MTU issues are a common cause of reliability problems for some Silverline routed customers. This article outlines the problem, and offers the two primary options to resolve this issue.
- Problem: PATH MTU Discovery
- Solution Option #1: IP TCP ADJUST-MSS - Requires implementation on your infrastructure.
- Solution Option #2: Allowing Fragmentation - Requires Silverline implementation.
- Alternatives to GRE Tunnels
Problem: End User Blocks the "PATH MTU Discovery" ICMP Response
"Path MTU Discovery" (PMTUD) is the Internet's way of packet size optimization when an Interface along the path is too small for a packet you've sent.
Normal packet size on the Internet is 1500 bytes. A GRE Tunnel encapsulates normal packets into another internet packet, and this Encapsulation takes up 64 bytes of data. Since a 1564 byte packet (1500 bytes for normal packet size PLUS the 64 bytes for GRE Tunnel Encapsulation) is too large for the GRE Tunnel's virtual interface, the router (in this case, Silverline's router) initiates "Path MTU Discovery," where the router responds to the end-user with an ICMP packet telling their interface to use a limit of 1436 bytes. (This number comes from 1500 - 64 = 1436.)
"Path MTU Discovery" is designed to alert the end user of this issue (so they can send smaller packets), but Path MTU Discovery doesn't function properly when the ICMP response is blocked by the end user's computer or network.
This is a well documented issue with Path MTU Discovery. See Further Reading for Cisco's documentation on PATH MTU Discovery.
Normal Traffic Path in Routed Silverline Setup (With Packets < 1436 bytes)
What Happens When End User Attempts to POST Packet Larger than 1436 Bytes.
1. End-user attempts to POST a packet that is larger than 1436 bytes.
2. When the Silverline router sees that the Packet is too large to fit inside the GRE Tunnel, the router (a) drops the packet and (b) sends an ICMP response with this router's interface MTU to the original sender (the end-user). Path MTU Discovery often works. However, a problem arises when end user computers or networks block the ICMP packets. In that case, the end user may never receive the Silverline router's ICMP packet.
3. If the end user doesn't receive the ICMP packet, they have no idea that they are over the packet limit. Thus, the end-user's computer will keep retransmitting data that is too large.
Solution Option #1: IP TCP ADJUST-MSS
Requires implementation on your infrastructure.
Solution: If you have a Cisco router upstream on the outbound/return route from your web server, a fix for the Path MTU Discovery issue is the Cisco command "ip tcp adjust-mss 1436" on the outbound traffic. Juniper added a similar capability in v14.2. This command should be applied on the interface(s) through which outbound Internet traffic is sent, i.e. facing your ISP. Note: This option is only applicable to TCP traffic. If there is UDP traffic, this option will not affect traffic.
Effect: This Cisco command re-sets the MTU on all packets transiting the router interface, telling the end user to use 1436, overwriting the Server's MSS. This ensures that the end user is sending packets that are small enough to fit down the GRE tunnel.
Solution Option #2: Allowing Fragmentation
Requires implementation on Silverline infrastructure.
Solution: If the TCP mss-adjust command is not an option, Silverline can configure customer GRE tunnels to allow fragmentation. Set MTU to 1476. Path MTU is still used.
Effects: This setting allows fragmented packets to pass down the GRE tunnels. Any POSTs larger than 1436 bytes are fragmented into packets that are 1436 bytes or smaller. When these fragmented packets reach your infrastructure, your router will have to reassemble them into the original sized packet.
Notes of Caution:
1) This may cause higher CPU usage on the terminating router.
2) Both fragmented packets and the reassembly of fragmented packets must be allowed by the router and firewalls.
How to Configure in Silverline Portal:
1. Navigate to Config > Routed Configuration > GRE Tunnel Management
2. Set up new GRE Tunnel as normal (see article on GRE Tunnel Setup) but be sure to check the "Fragmentation Allowed" checkbox, and set MTU to 1476.
Alternatives to GRE Tunnels
Due to the GRE PATH MTU Discovery issue and the inability for service providers to provide SLAs to GRE tunnel availability, customers may seek alternatives to GRE tunnels for clean traffic delivery. The alternatives are outlined in this article: Alternatives to GRE Tunnels.
More info in this Cisco.com Article: Resolve IPv4 Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPsec