How to Troubleshoot Black Hole Router Issues

This article defines the term "black hole" router, describes a method of locating black hole routers, and suggests three ways to avoid the data loss that can occur because of a black hole router.

On a TCP/IP-based wide area network (WAN), communication over some routes may fail if an intermediate network segment has a maximum packet size that is smaller than the maximum packet size of the communicating hosts--and if the router does not send an appropriate Internet Control Message Protocol (ICMP) response to this condition or if a firewall on the path drops such a response. Such a router is sometimes known as a "black hole" router.

You can locate a black hole router by using the Ping utility, which is a standard utility that is installed with the Microsoft Windows TCP/IP protocol. You can then use one of three methods of fixing or working around black hole routers.

When a network router receives a packet that is larger than the size of the Maximum Transmission Unit (MTU) of the next segment of a communications network, and that packet's IP layer "don't fragment" bit is flagged, the router is expected to send an ICMP "destination unreachable" message back to the sending host.

If the router does not send a message, the packet might be dropped, causing a variety of errors that vary with the program that is communicating over the unsuccessful link. (These errors do not occur if a program connects to a computer on a local subnet.) The behavior may seem intermittent, but closer examination shows that the behavior can be reproduced, for example, by having a client read a large file that is sent from a remote host.

Client-side Error

The client could not establish a connection to the remote computer. The most likely causes for this error are:

•	Remote connections cannot be enabled at the remote computer.
•	The maximum number of connections is exceeded at the remote computer.
•	A network error occurs while establishing the connection. Server-side Error: Event ID 1004 Source: TermService Description: "The terminal server cannot issue a client license. It was unable to issue the license due to a changed (mismatched) client license, insufficient memory, or an internal error. Further details for this problem may have been reported at the client's computer."

Locating a Black Hole Router

You can use the Ping utility to locate a black hole router, by setting the -f and -l parameters when you type the ping command.

•	The -f parameter causes the Ping utility to send an ICMP echo packet that has the IP "do not fragment" bit set.
•	The -l parameter sets the buffer, or payload, size of the ICMP echo packet. You specify this size by typing a number after the -l parameter.

The largest buffer that can be sent unfragmented is equal to the smallest MTU that exists along a route, minus the IP and ICMP headers (in other words, the smallest MTU minus 28). For example, Ethernet has an MTU of 1,500 bytes, so under the best circumstances, the Ping utility can echo an unfragmented packet, plus an ICMP buffer, of 1,472 bytes (1,500 minus 28). The syntax for the ping command in this case is: For all local IP addresses, the expected results are as follows:

•	If the MTU of every segment of a routed connection is at least 1,500, the packet is successfully returned.
•	If there are intermediate segments that have smaller MTUs, and the routers return the appropriate ICMP "destination unreachable" packet, the Ping utility displays the message, "Packet needs to be fragmented but DF set."
•	If there are intermediate segments that have smaller MTUs, and the routers do not return the appropriate ICMP "destination unreachable" packet, the Ping utility displays the message, "Request timed out."

By increasing the -l parameter on successive pings, you can identify how large an unfragmented packet can travel a specific route. The smallest MTU that is in general use is 576 bytes, so you can safely start with an ICMP buffer of 548 and then work up from there. For example, if the command Ping computer_name or IP_address -f -l 972 returns packets but Ping computer_name or IP_address -f -l 973 does not return packets, the largest MTU on that route is 1,000 (972 plus 28). The default MTUs of common network media are described in the following article in the Microsoft Knowledge Base:

Fixing or Working Around a Black Hole Router

WARNING: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use Registry Editor at your own risk.

The following three methods are ways to either fix or work around a black hole router.

Method 1

Enable PMTU Black Hole Detection on the Windows-based hosts that will be communicating over a WAN connection. Follow these steps:

1.	Start Registry Editor (Regedit.exe).
2.	Locate the following key in the registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tcpip\parameters
3.	On the Edit menu, click Add Value, and then add the following registry value: Value Name: EnablePMTUBHDetect Data Type: REG_DWORD Value: 1
4.	Quit Registry Editor, and then restart the computer.

Method 2

Configure intermediate routers to send ICMP Type 3 Code 4 messages ("destination unreachable, don't fragment (DF) bit sent and fragmentation required"). This might require a router software or firmware upgrade, router reconfiguration, or router replacement.

Method 3

Set the MTU of the host interface to be the largest size that the black hole router can handle, to guarantee that the largest possible packet size is sent over that connection. However, note that local traffic then uses smaller packets than necessary, as will traffic that uses the routed connections without problems.

This workaround assumes that you have identified the MTU and the state of all possible links that the host might use. After you identify the largest MTU size that is supported, manually set the MTU. Follow these steps:

1.	Click Start, and then click Control Panel.
2.	Double-click Network and Internet Connections, and then click to open the Network Connections folder.
3.	If more than one network connection is listed, for each connection, double-click the connection and then click the Support tab of the Status interface that opens. The connection that shows a Default Gateway entry is probably the network connection that is used to connect to the Internet. Note the name of the connection (for example, "Local Area Connection 2").
4.	Start Registry Editor (Regedit.exe).
5.	Under the HKEY_LOCAL_MACHINE tree, go to the following key: SYSTEM\CurrentControlSet\Control\Network\{4D36E972-E325-11CE-BFC1-08002BE10318}\
6.	Under that key are one or more keys that have numeric identifiers. Each of these keys has a Connection subkey. Examine each of the keys that look like this: ID_for_Adapter\Connection The Name value in the Connection subkey provides the network connection name that is used in the Network Connections folder. When you find the one that matches the name that you found in step 3, make a note of the ID_for_Adapter that the network connection name is under.
7.	Return to HKEY_LOCAL_MACHINE, and then locate the following key SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\ID_for_Adapter where ID_for_Adapter is the number that you noted in step 6. When you highlight this key, several values appear on the right side of the screen, including DefaultGateway and EnableDHCP.
8.	Right-click the right side of the screen, click New, and then click DWORD Value. Name the value MTU.
9.	Double-click the value so that you can edit the value, change Base to Decimal, and then enter the largest acceptable MTU size, which is the size that you identified by using the Ping tests.
10.	[ERIC NOTE]: For changes to take affect on an XP system, disable/enable the connection. Then test again with the ping. I also found you want to include the 28 bytes in the registry setting. IE: If your largest ping -l is 1300 then you want to set the MTU setting to 1328. (mturoute return 1328 (1300 +28). [ERIC NOTE]: So how do you test to see if the new MTU setting took affect? Lets say originally the server had no MTU setting set, you run mturoute and it determines MTU should be 1300. You set MTU to 1300, reboot, run mturout again and it still returns 1300. So did your MTU setting take place? Only way to really tell is change the MTU setting to something lower then 1300, set it to 1200 to test. If the MTU is set to 1200 then mturoute should report the max MTU is 1200. *****NOTE**** See comment above, this MTU setting only affect packets originating from the server, 1200 might be the largest packet you can send, but another host might be able to send you larger packets depending upon its MTU setting and equipment between. On VPN connections 1420 seems to typically be a good number :)
11.	Quit Registry Editor.

Note that if you still encounter problems with some servers, you might need to set the MTU lower than the Ping tests indicate because of other routers in that specific path. Repeatedly lower the MTU by 10 until access to those sites is successful.

For additional information manually setting the MTU, click the article number below to view the article in the Microsoft Knowledge Base:
For additional information, see Internet RFC 1191 and RFC