Most network engineers and sysadmins would probably say that they're intimately familiar with 'traceroute', and consider it one of their fundamental network troubleshooting tools... I certainly do. But you might be amazed to learn, as I did, how much you don't know about traceroute.
Among other things, this presentation shows you:
One of the coolest tricks I learned from this presentation is, to find out more about what's at the other end of some hop that appears to be a point-to-point link, assume that the IP address you see is one of the two addresses in a /30 subnet (as is commonly assigned to point-to-point links), and do a DNS reverse lookup of the other address in the /30.
This is useful, for example, in figuring out which egress port a packet went out on, since traceroute normally only shows you the ingress ports for each device along the way. For example, let's say I was looking at the following traceroute output, and wanted to know the egress port on router #3, as the packet moved to router #4:
brent% traceroute www.google.com traceroute: Warning: www.google.com has multiple addresses; using 184.108.40.206 traceroute to google.navigation.opendns.com (220.127.116.11), 64 hops max, 40 byte packets 1 192.168.0.1 (192.168.0.1) 3.145 ms 2.573 ms 2.382 ms 2 75-101-29-1.dsl.static.sonic.net (18.104.22.168) 9.555 ms 9.054 ms 9.089 ms 3 127.at-X-X-X.gw3.200p-sf.sonic.net (22.214.171.124) 9.510 ms 9.871 ms 9.194 ms 4 200.ge-0-1-0.gw.equinix-sj.sonic.net (126.96.36.199) 11.965 ms 11.870 ms 11.839 ms 5 0.as0.gw2.equinix-sj.sonic.net (188.8.131.52) 11.928 ms 12.519 ms 12.394 ms 6 GigabitEthernet3-1.GW2.SJC7.ALTER.NET (184.108.40.206) 11.360 ms 16.257 ms 11.268 ms 7 0.so-0-0-1.XL4.SJC7.ALTER.NET (220.127.116.11) 11.729 ms 11.679 ms 11.403 ms 8 0.so-7-0-0.XL2.PAO1.ALTER.NET (18.104.22.168) 14.775 ms 17.455 ms 0.so-5-0-0.XL2.PAO1.ALTER.NET (22.214.171.124) 15.548 ms 9 POS7-0.GW6.PAO1.ALTER.NET (126.96.36.199) 12.886 ms 13.143 ms 13.029 ms 10 188.8.131.52 (184.108.40.206) 13.517 ms 14.708 ms 16.566 ms 11 * * * 12 * * * ^C
To find out more about router #3's egress port, I look at the IP address for router #4 (220.127.116.11), figure out what would be the other IP address in the same /30 (18.104.22.168; hint: the lower address in a /30 pair always ends in an odd number, and the higher address always ends in an even number, so if the address you know ends in an odd number, the other address in the same /30 is going to be the next higher number, and if the address you know is even, the other is going to be the next lower number), and do a DNS reverse lookup of that address:
brent% dig -x 22.214.171.124 ; <<>> DiG 9.4.3-P3 <<>> -x 126.96.36.199 ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49382 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;188.8.131.52.in-addr.arpa. IN PTR ;; ANSWER SECTION: 184.108.40.206.in-addr.arpa. 259200 IN PTR 200.ge-6-3-0.gw3.200p-sf.sonic.net. ;; Query time: 31 msec ;; SERVER: 220.127.116.11#53(18.104.22.168) ;; WHEN: Fri Nov 13 09:42:05 2009 ;; MSG SIZE rcvd: 91
Another handy tip from the presentation is that, since light travels through fiber optic cable at about 200 km (or 125 miles, if you prefer) per millisecond, each 1 ms of delay shown by traceroute (which, remember, is round trip delay) should represent about 100 km (62.5 mi) of distance if the delay were due entirely to the distance travelled (i.e., no queuing or processing delays). Using that fact, you can see that 40ms for a packet to go from San Francisco to New York (about 2500 miles, or 4000km) would be "normal", but 40ms for a packet to go from San Francisco to San Jose (about 50 miles, or 80km) would indicate a problem; it should take the packet less than 1ms to cover that distance and back, so something else (congestion or processing delays, for example) must account for the other 39ms.
There's a lot more in this presentation, about more complex issues such as
Anyway, if you ever use traceroute, I highly recommend that you review this excellent presentation. I think you'll be pleasantly surprised at how much you learn.
Thanks to Strata Chalup of Virtual.net for bringing this very informative presentation to my attention.