Troubleshooting Tools — Finding Where a Connection Breaks

What you'll learn

• A method that finds the broken step quickly instead of guessing: link, address, route, name, service
• How to read ping: loss, round-trip times, duplicates and "unreachable" messages
• How traceroute discovers the path with the TTL field, and why * * * often means nothing is wrong
• How to check names with dig and nslookup, and services with ss (or the older netstat)
• How to capture and read packets with tcpdump and Wireshark, including the difference between "refused" and "no answer"
• You will build three tools: a ping summariser, a traceroute reader and a handshake detector for tcpdump output

⚠️ Ethics first: capture and probe only your own devices and networks, or ones you have written permission to test. A packet capture can contain other people's private data, and intercepting traffic on someone else's network is illegal in most countries. Every sample in this lesson is made up, with addresses from the documentation ranges.

The plumber's method 🔧

When water stops coming out of a tap, a good plumber does not tear down the wall. She checks the tap, then the pipe under the sink, then the valve, then the main supply, one section at a time, in order, until she finds the section where water goes in but does not come out.

A network connection is a pipe made of layers, and each layer needs the one below it. So check them bottom up, and stop at the first one that fails:

1. Link      is the cable plugged in / Wi-Fi associated?   ip link
2. Address   do I have an IP address and a gateway?         ip addr, ip route
3. Route     can I reach the gateway, then the far side?    ping, traceroute
4. Name      does the name resolve to the right address?    dig, nslookup
5. Service   is the program listening, and is it allowed?   ss, curl -v, nc -vz
   ... and when the tools disagree: look at the packets     tcpdump, Wireshark

Two habits make the method work. Change one thing at a time and write down what you saw, so you know which change fixed it. And know what "good" looks like: run these tools on a healthy network first, so that an odd result stands out.

Steps 1 to 3: link, address, and ping

Start with ip link: an interface marked state DOWN or NO-CARRIER has no working link (unplugged cable, Wi-Fi not associated, disabled port). Then ip addr: do you have an address? An IPv4 address in 169.254.0.0/16 is a link-local address the system gave itself (RFC 3927) because DHCP failed. Then ip route: is there a default via ... line?

Now ping the gateway, then something beyond it. Ping sends ICMP echo requests (type 8) and waits for echo replies (type 0), as defined in RFC 792:

$ ping -c 4 192.0.2.1
PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.
64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=0.512 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=0.498 ms
64 bytes from 192.0.2.1: icmp_seq=3 ttl=64 time=0.731 ms
64 bytes from 192.0.2.1: icmp_seq=4 ttl=64 time=0.455 ms

--- 192.0.2.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 0.455/0.549/0.731/0.107 ms

How to read it:

• icmp_seq numbers each request. A gap in the sequence is a lost packet (or a lost reply). A reply that arrives very late still counts; it just has a huge time.
• time is the round-trip time. Compare it with what is normal for that path: under a millisecond on a LAN, tens of milliseconds across a country. A large mdev (the spread) means the delay jumps around (jitter), which hurts calls and games.
• ttl is what is left of the reply's TTL. Systems start at different values (Linux commonly 64, Windows 128), so a reply with ttl=54 crossed about ten routers on the way back.
• (DUP!) marks a duplicate reply: the same packet arrived twice, a sign of a loop or a misbehaving device. Ping reports them separately as +N duplicates.
• From 192.168.1.10 icmp_seq=1 Destination Host Unreachable coming from your own address means your host could not even find the target's MAC address with ARP: the target is on your subnet but not answering. Ping counts these as +N errors.

One warning: many servers and firewalls block ICMP on purpose. No reply to ping does not prove a host is down, and a reply does not prove its web server works. Ping tests reachability at layer 3, nothing more.

Step 3, continued: traceroute

Ping says "the far side is (not) reachable". traceroute says where the packets go, hop by hop. It uses the TTL rule you learned in the routing lesson: every router decrements the TTL, and the router that brings it to 0 drops the packet and sends back ICMP time exceeded (type 11), revealing its address.

So traceroute sends probes with TTL 1, then TTL 2, then 3, and so on, usually three probes per hop. The first router answers the TTL 1 probes, the second router the TTL 2 probes, and when a probe finally reaches the destination, the destination answers instead: Linux traceroute sends UDP probes to high ports (from 33434 up), so the destination replies port unreachable; Windows tracert and traceroute -I use ICMP echo, so it replies with an echo reply; traceroute -T uses TCP SYN packets, which pass many firewalls.

$ traceroute -n 198.51.100.20
traceroute to 198.51.100.20 (198.51.100.20), 30 hops max, 60 byte packets
 1  192.168.1.1     0.812 ms  0.701 ms  0.655 ms     <- your router
 2  10.10.0.1       8.102 ms  7.950 ms  8.311 ms     <- the ISP
 3  * * *                                            <- a router that does not answer
 4  203.0.113.9    15.214 ms 14.902 ms 15.050 ms
 5  198.51.100.20  16.127 ms 16.301 ms 15.840 ms     <- the destination

Reading it takes care:

• * * * means "no answer within the timeout". If later hops answer, that router forwards traffic fine; it just does not send ICMP, or rate-limits it. It is not the problem.
• If the stars continue until the hop limit (30 by default), the trail goes cold after the last hop that answered: either the path is broken there, or the destination (or a firewall in front of it) drops the probes. Try another probe type before blaming the network.
• A jump in latency that stays for every later hop shows a long or busy link (a long-distance cable, a congested uplink). A single slow hop whose successors are fast is a router that answers ICMP slowly while forwarding normally.
• The return path can differ from the forward path, so times include a path you cannot see. mtr combines ping and traceroute and keeps updating loss and delay per hop, which is better for intermittent problems.

Steps 4 and 5: the name and the service

The name. If ping 192.0.2.80 works but ping www.example.com fails, the problem is DNS. Ask directly and compare resolvers:

$ dig +short www.example.com              # your configured resolver
$ dig @192.0.2.53 www.example.com         # a specific server: same answer?
$ nslookup www.example.com                # also available on Windows
$ getent hosts www.example.com            # what programs see, /etc/hosts included

resolvectl status (on systems with systemd-resolved) or /etc/resolv.conf tells you which resolver you are using. Read the status as in the DNS lesson: NXDOMAIN is a wrong or missing name, SERVFAIL a broken zone or DNSSEC failure, and a timeout means you cannot reach the resolver at all. A stale entry in /etc/hosts overrides DNS, a classic source of "it works on my machine".

The service. The name resolves and the host answers, but the application still fails. On the server, check that the program listens, on which address, and which process owns the socket:

$ ss -tlnp
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
LISTEN  0       511           0.0.0.0:80          0.0.0.0:*      users:(("nginx",pid=901,fd=6))
LISTEN  0       4096        127.0.0.1:5432        0.0.0.0:*      users:(("postgres",pid=812,fd=6))

The flags mean TCP (-t), listening (-l), numeric (-n) and processes (-p); ss -tn shows established connections. The local address matters: 0.0.0.0 (or [::]) accepts connections on every interface, while 127.0.0.1 only accepts them from the same machine. Here PostgreSQL cannot be reached from another host at all, by design or by mistake. On older systems netstat -tlnp gives the same information, and on Windows netstat -ano.

From the client, test the port itself: nc -vz 192.0.2.80 443 or curl -v https://www.example.com/. Remember the TCP lesson: "connection refused" means the host answered with RST (nothing listens on that port), while a timeout means nothing answered (a firewall dropping packets, a wrong address or a broken route).

When the tools disagree: read the packets

Sometimes every tool says "fine" and the application still fails. Then you look at what actually crosses the wire with tcpdump (command line, available on almost every server) or Wireshark (graphical, with decoders for hundreds of protocols). Both read and write the same .pcap files, so a common workflow is to capture on a server with tcpdump -w capture.pcap and open the file in Wireshark on your laptop.

Filters keep captures small and readable, and the two tools use different filter languages:

• Capture filters (BPF syntax, used by tcpdump and in Wireshark's capture dialog) decide what is recorded at all: tcpdump -i eth0 -n host 192.0.2.10 and port 443.
• Display filters (Wireshark only) hide packets that were already captured: ip.addr == 192.0.2.10 && tcp.port == 443, dns, tcp.flags.syn == 1 && tcp.flags.ack == 0 (connection attempts) or tcp.analysis.retransmission.

A tcpdump line, read with -n so no names are looked up:

12:00:00.000000 IP 192.0.2.10.51514 > 203.0.113.80.443: Flags [S], seq 1000, win 64240, length 0
12:00:00.023412 IP 203.0.113.80.443 > 192.0.2.10.51514: Flags [S.], seq 5000, ack 1001, win 65160, length 0
12:00:00.023500 IP 192.0.2.10.51514 > 203.0.113.80.443: Flags [.], ack 5001, win 502, length 0
12:00:00.030000 IP 192.0.2.10.51520 > 203.0.113.80.22: Flags [S], seq 77, win 64240, length 0
12:00:00.030950 IP 203.0.113.80.22 > 192.0.2.10.51520: Flags [R.], seq 0, ack 78, win 0, length 0

Each line is: timestamp, protocol, source.port > destination.port, then the TCP flags: S SYN, F FIN, P push, R RST, and . for ACK. So [S] is a connection attempt, [S.] the SYN-ACK, [.] a plain ACK and [R.] a reset. The first three lines are a complete handshake to port 443 (the SYN-ACK came back after 23.4 ms); the last two are a refusal from port 22. A third pattern is the one behind timeouts: the same SYN repeated after about 1, 2 and 4 seconds (the retransmission timeout doubling each time) with nothing coming back. With TLS you will not see the content of the conversation, only who talked to whom, when and how much, which is often enough to find the fault.

The method in practice

"The website does not load." ip link shows the interface up, ip addr shows 192.168.1.10/24, and ip route has a default route. ping 192.168.1.1 works: link, address and gateway are fine. ping 192.0.2.80 (the site's address) works too, so routing is fine. dig www.example.com returns SERVFAIL, and dig @192.0.2.53 www.example.com with another resolver returns the right address. The fault is your resolver, not the site: switch resolvers or report it, and stop looking at cables.

"The app cannot reach the database." On the database server, ss -tlnp shows PostgreSQL on 127.0.0.1:5432, so it only accepts local connections. From the app server, nc -vz 10.0.0.5 5432 says "connection refused" (a reply: the host is up, nothing listens on that address). Fix the listen address, then check that the firewall allows port 5432 from the app server only.

"Some uploads hang." Ping and small requests work, but large transfers stall. A capture shows big segments sent again and again with no ACK. Small packets pass and large ones vanish: a classic MTU problem, often on a VPN or tunnel where something blocks the ICMP messages that would tell the sender to use smaller packets.

Each case followed the same path: bottom up, one test per layer, and a packet capture only when the simple tools could not explain what was happening.

Recap

• Troubleshoot bottom up: link (ip link), address (ip addr, ip route), route (ping, traceroute), name (dig, nslookup), service (ss, curl -v, nc -vz). Change one thing at a time.
• ping: gaps in icmp_seq are loss, time and mdev show delay and jitter, DUP! means duplicates, and no reply does not prove the host is down.
• traceroute uses TTL expiry; * * * followed by answering hops is harmless; a delay that stays from one hop onward shows a long or busy link.
• ss shows what listens and on which address: 127.0.0.1 is local only. "Refused" means RST came back; a timeout means nothing did.
• tcpdump and Wireshark show the truth on the wire: [S], [S.], [.], [R.]. Capture only where you are allowed to.

Practice next: on your own machine, run ping -c 5 to your router, traceroute -n to a site you use, and ss -tlnp, and explain every line. Then capture your own DNS queries with sudo tcpdump -n -i any port 53 while you browse. Next lesson: Wi-Fi, VPNs and firewalls.