This article defines the term "black hole" router,
describes a method of locating black hole routers, and suggests three ways to
avoid the data loss that can occur because of a black hole router.
On a TCP/IP-based wide area network (WAN), communication
over some routes may fail if an intermediate network segment has a maximum
packet size that is smaller than the maximum packet size of the communicating
hosts--and if the router does not send an appropriate Internet Control Message
Protocol (ICMP) response to this condition or if a firewall on the path drops such a response. Such a router is sometimes known as
a "black hole" router.
You can locate a black hole router by using
the Ping utility, which is a standard utility that is installed with the
Microsoft Windows TCP/IP protocol. You can then use one of three methods of
fixing or working around black hole routers.
When a network router
receives a packet that is larger than the size of the Maximum Transmission Unit
(MTU) of the next segment of a communications network, and that packet's IP
layer "don't fragment" bit is flagged, the router is expected to send an ICMP
"destination unreachable" message back to the sending host.
router does not send a message, the packet might be dropped, causing a variety
of errors that vary with the program that is communicating over the
unsuccessful link. (These errors do not occur if a program connects to a
computer on a local subnet.) The behavior may seem intermittent, but closer
examination shows that the behavior can be reproduced, for example, by having a
client read a large file that is sent from a remote host.
The client could not establish a
connection to the remote computer. The most likely causes for this error are:
|•||Remote connections cannot be enabled at the remote
|•||The maximum number of connections is exceeded at the
|•||A network error occurs while establishing the connection.
Server-side Error: Event ID 1004
Description: "The terminal server cannot issue a client license. It was
unable to issue the license due to a changed (mismatched) client license,
insufficient memory, or an internal error. Further details for this problem may
have been reported at the client's computer."
Locating a Black Hole Router
You can use the Ping utility to locate a black hole router, by
setting the -f
parameters when you type the ping
|•||The -f parameter causes the Ping utility to send an ICMP echo packet
that has the IP "do not fragment" bit set. |
|•||The -l parameter sets the buffer, or payload, size of the ICMP echo
packet. You specify this size by typing a number after the -l parameter.|
The largest buffer that can be sent unfragmented is equal to
the smallest MTU that exists along a route, minus the IP and ICMP headers (in
other words, the smallest MTU minus 28). For example, Ethernet has an MTU of
1,500 bytes, so under the best circumstances, the Ping utility can echo an
unfragmented packet, plus an ICMP buffer, of 1,472 bytes (1,500 minus 28). The
syntax for the ping
command in this case is:
ping computer_name or IP_address -f -l 1472
For all local IP addresses, the expected results are as follows:
|•||If the MTU of every segment of a routed connection is at
least 1,500, the packet is successfully returned.|
|•||If there are intermediate segments that have smaller MTUs,
and the routers return the appropriate ICMP "destination unreachable" packet,
the Ping utility displays the message, "Packet needs to be fragmented but DF
|•||If there are intermediate segments that have smaller MTUs,
and the routers do not return the appropriate ICMP "destination unreachable"
packet, the Ping utility displays the message, "Request timed out."|
By increasing the -l
parameter on successive pings, you can identify how large an
unfragmented packet can travel a specific route. The smallest MTU that is in
general use is 576 bytes, so you can safely start with an ICMP buffer of 548
and then work up from there. For example, if the command Ping
computer_name or IP_address -f -l 972
returns packets but Ping computer_name or
IP_address -f -l 973
does not return packets, the
largest MTU on that route is 1,000 (972 plus 28). The default MTUs of common
network media are described in the following article in the Microsoft Knowledge
Default MTU Size for Different Network Topology
[Eric] Here is a great utility, MTUROUTE.exe
, to do the ping commands described above, but faster.
The mturoute command will include the extra 28 bytes of TCP header info. IE: If your ping -f -l 1472 is the highest MTU then mturoute will return 1500 (1472 + 28)
****** BIG NOTE *****
Testing results can vary depending on where you run the commands, IE: Source or target. On a Windows server the MTU setting
on the system you run the command will set the maximum MTU you can send, but you can receive larger MTU'sfs. For example, lest say Server A has an MTU set to 1300 and
Server B is set to 1500. If you ping from server A to server B, the largest packet will be 1300, however if you ping
fron server B to server A the largest packet will be 1500.
Fixing or Working Around a Black Hole RouterWARNING
: If you use Registry Editor incorrectly, you may cause serious
problems that may require you to reinstall your operating system. Microsoft
cannot guarantee that you can solve problems that result from using Registry
Editor incorrectly. Use Registry Editor at your own risk.
The following three methods are ways to either fix
or work around a black hole router.
Enable PMTU Black Hole Detection on the Windows-based hosts that
will be communicating over a WAN connection. Follow these steps:
|1.||Start Registry Editor (Regedit.exe).|
|2.||Locate the following key in the registry:|
|3.||On the Edit menu, click Add Value, and then add the following registry value:|
Value Name: EnablePMTUBHDetect
Data Type: REG_DWORD
|4.||Quit Registry Editor, and then restart the
Configure intermediate routers to send ICMP Type 3 Code 4
messages ("destination unreachable, don't fragment (DF) bit sent and
fragmentation required"). This might require a router software or firmware
upgrade, router reconfiguration, or router replacement.
Set the MTU of the host interface to be the largest size that the
black hole router can handle, to guarantee that the largest possible packet
size is sent over that connection. However, note that local traffic then uses
smaller packets than necessary, as will traffic that uses the routed
connections without problems.
This workaround assumes that you have
identified the MTU and the state of all possible links that the host might use.
After you identify the largest MTU size that is supported, manually set the
MTU. Follow these steps:
|1.||Click Start, and then click Control Panel.|
|2.||Double-click Network and Internet
Connections, and then click to open the Network Connections folder.|
|3.||If more than one network connection is listed, for each
connection, double-click the connection and then click the Support tab of the Status interface that opens. The connection that shows a Default Gateway entry is probably the network connection that is used to connect
to the Internet. Note the name of the connection (for example, "Local Area
|4.||Start Registry Editor (Regedit.exe).|
|5.||Under the HKEY_LOCAL_MACHINE tree, go to the following key:|
|6.||Under that key are one or more keys that have numeric
identifiers. Each of these keys has a Connection subkey. Examine each of the keys that look like this: |
ID_for_Adapter\Connection The Name value in the Connection subkey provides the network connection name that is used in the
Network Connections folder. When you find the one that matches the name that
you found in step 3, make a note of the
ID_for_Adapter that the network connection name is
|7.||Return to HKEY_LOCAL_MACHINE, and then locate the following key|
SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\ID_for_Adapter where ID_for_Adapter is
the number that you noted in step 6. When you highlight this key, several
values appear on the right side of the screen, including DefaultGateway and EnableDHCP.
|8.||Right-click the right side of the screen, click New, and then click DWORD Value. Name the value MTU.|
|9.||Double-click the value so that you can edit the value,
change Base to Decimal, and then enter the largest acceptable MTU size, which is the
size that you identified by using the Ping tests.|
[ERIC NOTE]: For changes to take affect on an XP system, disable/enable the connection. Then test again with the ping.
I also found you want to include the 28 bytes in the registry setting. IE: If your largest ping -l is 1300 then you
want to set the MTU setting to 1328. (mturoute return 1328 (1300 +28).
[ERIC NOTE]: So how do you test to see if the new MTU setting took affect? Lets say originally the server had no MTU setting set, you
run mturoute and it determines MTU should be 1300. You set MTU to 1300, reboot, run mturout again and it still returns 1300. So did
your MTU setting take place? Only way to really tell is change the MTU setting to something lower then 1300, set it to 1200 to test.
If the MTU is set to 1200 then mturoute should report the max MTU is 1200. *****NOTE**** See comment above, this MTU setting only
affect packets originating from the server, 1200 might be the largest packet you can send, but another host might be able to send
you larger packets depending upon its MTU setting and equipment between.
On VPN connections 1420 seems to typically be a good number :)
|11.||Quit Registry Editor.|
Note that if you still encounter problems with some servers,
you might need to set the MTU lower than the Ping tests indicate because of
other routers in that specific path. Repeatedly lower the MTU by 10 until
access to those sites is successful.
For additional information manually setting the MTU, click the
article number below to view the article in the Microsoft Knowledge Base:
TCP/IP and NBT Configuration Parameters for Windows XP
For additional information, see Internet RFC 1191 and RFC