Digital Developer Conference on Cloud Native Security: Register for free and choose your sessions. June 24, 25, & July 1, 2020 Learn more

Learn Linux 101: Basic network troubleshooting

Overview

In this tutorial, learn to troubleshoot networking issues on your Linux client system. Learn to:

  • Manually configure network interfaces, including viewing and changing the configuration of network interfaces using iproute2.
  • Manually configure routing, including viewing and changing routing tables and setting the default route using iproute2.
  • Debug problems associated with the network configuration.
  • Recognize the legacy net-tools commands.

Networking in Linux

In today’s world, computer networking enables information sharing, research, and commerce across the country or across the world. Research Kenyan national parks. Sure. Buy a Swiss train ticket. No problem. Email someone down the street or on another continent. See photos from outer space. Computer networking makes all this and much more possible. But what happens when things don’t work?

This tutorial helps you prepare for Objective 109.3 in Topic 109 of the Linux Administrator (LPIC-1) exam 101. The objective has a weight of 4. This tutorial reflects the Version 5.0 objectives as updated on October 29, 2018.

Prerequisites

To get the most from the tutorials in this series, you need a basic knowledge of Linux and a working Linux system on which you can practice the commands that are covered in this tutorial. For this tutorial, you will also need one or more network connections. Sometimes, different versions of a program format output differently. So, your results might not always look exactly like the listings and figures that are shown here.

The examples in this tutorial come from Fedora 31, and Ubuntu 18.04.3 LTS.

Networking tool sets

For many users, networking configuration just happens automatically when you plug in a network cable or provide some kind of login credentials to a wifi network. In our tutorial, “Learn Linux 101: Persistent network configuration“, I showed you basic network configuration using graphical tools when you need to do some manual configuration. In this tutorial, I will show you the command line tools that you use for these tasks and for diagnosing network problems. For the LPI certification, you need to use and understand the iproute2 set of networking commands. These were developed to better integrate with the 2.2 kernel and have been around for a long time now. The older net-tools suite is still in use as many older Linux people know them and have them ingrained. Despite their modern shortcomings they still perform many useful functions so you will probably work with people who don’t know the newer tools. As early as 2009, there was a discussion about either removing net-tools altogether or not including it with distributions. This discussion is ongoing.

On some systems, such as Fedora, the iproute2 tools are installed from the iproute (no 2) package, while on Ubuntu the package is called iproute2. Listing 1 shows how to check which packages are installed or available using the dnf command on Fedora or the apt command on Ubuntu.

Listing 1. Checking for networking tools packages
ian@ianattic5-f31 ~]$ # Fedora 31
[ian@ianattic5-f31 ~]$ dnf list iproute iproute2 net-tools
Last metadata expiration check: 0:24:28 ago on Tue 24 Mar 2020 09:03:02 PM EDT.
Installed Packages
iproute.x86_64                 5.4.0-1.fc31                             @updates
net-tools.x86_64               2.0-0.55.20160912git.fc31                @fedora

ian@attic4-u18:~$ # Ubuntu 18.04 LTS
ian@attic4-u18:~$ apt list iproute iproute2 net-tools
Listing... Done
iproute2/bionic,now 4.15.0-2ubuntu1 amd64 [installed]
net-tools/bionic,now 1.60+git20161116.90da8a0-1ubuntu1 amd64 [installed]

The iproute or iproute2 packages install a number of binaries which you can list using the dnf, rpm, or dpkg tools. Listing 2 shows how to list the installed binaries on Ubuntu.

Listing 2. Binary files installed by iproute2
ian@attic4-u18:~$ dpkg -L  iproute2 | grep "bin/"
/bin/ip
/bin/ss
/sbin/bridge
/sbin/devlink
/sbin/rtacct
/sbin/rtmon
/sbin/tc
/sbin/tipc
/usr/bin/lnstat
/usr/bin/nstat
/usr/bin/rdma
/usr/bin/routef
/usr/bin/routel
/usr/sbin/arpd
/usr/sbin/genl
/sbin/ip
/usr/bin/ctstat
/usr/bin/rtstat

This tutorial focuses on the ip command for configuration of interfaces and routes. Other tools that you learn about in this tutorial come from the following packages:

  • hostname
  • traceroute or inetutils-traceroute
  • iputils or iputils-ping or iputils-tracepath
  • nmap-ncat or netcat-openbsd

The names and contents of these packages differ between rpm and deb based systems. So, this is just a general guide to where you might look for these commands.

Introduction to using the ip command

Before I start changing things using ip, I’ll show you how to view information about your existing network.

The ip command has the following general form:

ip [ OPTIONS ] OBJECT { COMMAND | help }

Invoked without options, you will get brief help for the command as shown in Listing 3.

Listing 3. Help for the ip command
ian@ianattic5-f31 ~]$ ip
Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
       ip [ -force ] -batch filename
where  OBJECT := { link | address | addrlabel | route | rule | neigh | ntable |
                   tunnel | tuntap | maddress | mroute | mrule | monitor | xfrm |
                   netns | l2tp | fou | macsec | tcp_metrics | token | netconf | ila |
                   vrf | sr | nexthop }
       OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
                    -h[uman-readable] | -iec | -j[son] | -p[retty] |
                    -f[amily] { inet | inet6 | mpls | bridge | link } |
                    -4 | -6 | -I | -D | -M | -B | -0 |
                    -l[oops] { maximum-addr-flush-attempts } | -br[ief] |
                    -o[neline] | -t[imestamp] | -ts[hort] | -b[atch] [filename] |
                    -rc[vbuf] [size] | -n[etns] name | -N[umeric] | -a[ll] |
                    -c[olor]}

Otherwise, you need to specify some networking object to operate on. Examples include link, address, route, or neighbor. Most objects can be written in full form or in an abbreviated form. If you are in doubt about an abbreviation, the shortest unambiguous form usually works.

The command is the action to be performed on the object. Commands vary somewhat by object type. You can usually add, delete, and show (or list) objects but not all objects support all of these commands. The help for an object tells you the supported commands. For example, use ip link help or ip neigh help to get help for links or neighbors. Listing 4 shows the help for neighbors, with n being a sufficient abbreviation for neigh. Note that neighbor or neighbour will also work.

Listing 4. Help for the ip neighbor object
[ian@ianattic5-f31 ~]$ ip n help
Usage: ip neigh { add | del | change | replace }
        { ADDR [ lladdr LLADDR ] [ nud STATE ] | proxy ADDR } [ dev DEV ]
        [ router ] [ extern_learn ] [ protocol PROTO ]

    ip neigh { show | flush } [ proxy ] [ to PREFIX ] [ dev DEV ] [ nud STATE ]
        [ vrf NAME ]

STATE := { permanent | noarp | stale | reachable | none |
           incomplete | delay | probe | failed }

If you invoke ip OBJECT for some object, you will get a summary of information for the available objects of that type. Continuing with the neighbor example you might see something as shown in Listing 5.

Listing 5. Display neighbors using ip command
ian@attic4-u18:~$ ip n
192.168.1.38 dev enp2s0 lladdr ac:18:26:6c:48:aa REACHABLE
192.168.1.37 dev enp2s0 lladdr 00:1b:a9:8a:18:91 STALE
192.168.1.1 dev enp2s0 lladdr 1c:f2:9a:c7:50:0d REACHABLE
192.168.1.25 dev enp2s0 lladdr 30:5a:3a:08:da:e4 REACHABLE
2606:a000:1126:c8a8:5a1e:d3c4:ffe:3894 dev enp2s0 lladdr 30:5a:3a:08:da:e4 STALE
fe80::bc87:36ff:fe3f:c4b2 dev enp2s0 lladdr 1c:f2:9a:c7:50:0d router DELAY
fe80::efaa:4539:a6ba:27b9 dev enp2s0 lladdr 30:5a:3a:08:da:e4 STALE

This information comes from the Neighbor Discovery Cache (NDC). Reachable locations become stale if there is no traffic for 30 seconds.

Detailed man pages for the ip command

You can find more detailed help for using the ip command with various objects in the man or info pages using a command such as man ip-neighbour or info ip-link. The available pages are:

  • ip-address
  • ip-addrlabel
  • ip-l2tp
  • ip-link
  • ip-mad‐dress
  • ip-monitor
  • ip-mroute
  • ip-neighbour
  • ip-netns
  • ip-ntable
  • ip-route
  • ip-rule
  • ip-tcp_metrics
  • ip-token
  • ip-tunnel

Note that abbreviations and alternate spellings such as neighbor for neighbour are not supported for displaying these extended manual pages and also there are no individual commands with these names.

Viewing interfaces, addresses, and routes using the ip command

In order to transmit packets over an interface or link, you need a link, at least one IP address associated with the link, and a routing table that tells you which link to transmit packets over to a particular destination. This is a simplistic view of basic networking, but it will serve for learning more about the ip command. Listing 6 shows a basic example from my Ubuntu system.

ian@attic4-u18:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen
   1000 link/ether 84:16:f9:04:7a:2a brd ff:ff:ff:ff:ff:ff
3: enp4s0: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:23:54:33:f3:ee brd ff:ff:ff:ff:ff:ff

This system has a loopback link (lo) and two Ethernet links (enp2s0 and enp4s0). The local loopback link and the link enp2s0 are both up while the link enp4s0 is down (does not show as being UP). The status of LOWER_UP indicates that the underlying media, such as an Ethernet cable, is connected.

Notice that this information does not show any traffic statistics for the links. The ip command has a number of options, including -s (or -stats, or -statistics) to display statistics, or -br (or -brief) to display only basic information. You can also restrict information to IP version 6 (-6 option) or IP version 4 (-4 option). These options may not apply to all objects. Find out more about the available options using the man or info pages (man ip or info ip). Listing 7 shows two examples of these options.

Listing 7. Using options with the ip command
ian@attic4-u18:~$ ip -br link show
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp2s0           UP             84:16:f9:04:7a:2a <BROADCAST,MULTICAST,UP,LOWER_UP>
enp4s0           DOWN           00:23:54:33:f3:ee <BROADCAST,MULTICAST>
ian@attic4-u18:~$ ip -s link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast
    12434028   175440   0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    12434028   175440   0       0       0       0
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen
    1000 link/ether 84:16:f9:04:7a:2a brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    181493549  740195   0       0       0       39546
    TX: bytes  packets  errors  dropped carrier collsns
    10908959   121827   0       0       0       0
3: enp4s0: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:23:54:33:f3:ee brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    47979      118      0       0       0       6
    TX: bytes  packets  errors  dropped carrier collsns
    29955049   165003   0       0       0       0

The link information does not tell you whether there is an IPv4 or IPv6 address associated with any of the links.

Listing 8 shows how to use the ip command with the addr object to see the IP addresses associated with your links. Notice that link enp4s0 does not have any IP address associated with it. This link is currently configured to acquire addresses using Dynamic Host Configuration Protocol (DHCP).

Listing 8. Using ip addr
iian@attic4-u18:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 84:16:f9:04:7a:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.24/24 brd 192.168.1.255 scope global dynamic noprefixroute enp2s0
       valid_lft 83404sec preferred_lft 83404sec
    inet6 2606:a000:1126:c8a8:d517:956c:66ed:b709/64 scope global temporary dynamic
       valid_lft 86356sec preferred_lft 27375sec
    inet6 2606:a000:1126:c8a8:f419:b701:db6f:8608/64 scope global temporary deprecated dynamic
       valid_lft 86356sec preferred_lft 0sec
    inet6 2606:a000:1126:c8a8:b1:978b:6813:4705/64 scope global temporary deprecated dynamic
       valid_lft 86356sec preferred_lft 0sec
    inet6 2606:a000:1126:c8a8:a96e:36bd:8f26:5aca/64 scope global temporary deprecated dynamic
       valid_lft 86356sec preferred_lft 0sec
    inet6 2606:a000:1126:c8a8:6da4:fb09:dca6:8ffa/64 scope global temporary deprecated dynamic
       valid_lft 86356sec preferred_lft 0sec
    inet6 2606:a000:1126:c8a8:5d0:831b:2c28:9d40/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86356sec preferred_lft 86356sec
    inet6 fe80::48bc:4691:9ac0:d027/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: enp4s0: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 00:23:54:33:f3:ee brd ff:ff:ff:ff:ff:ff

As with other forms of the ip command you can use options to change output. For example, -br for brief output or -4 or -6 to limit output to IPv4 or IPv6 respectively. Listing 9 shows two examples.

Listing 9. Limiting ip addr output
ian@attic4-u18:~$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
enp2s0           UP             192.168.1.24/24 2606:a000:1126:c8a8:d517:956c:66ed:b709/64
2606:a000:1126:c8a8:f419:b701:db6f:8608/64 2606:a000:1126:c8a8:b1:978b:6813:4705/64
2606:a000:1126:c8a8:a96e:36bd:8f26:5aca/64 2606:a000:1126:c8a8:6da4:fb09:dca6:8ffa/64
2606:a000:1126:c8a8:5d0:831b:2c28:9d40/64 fe80::48bc:4691:9ac0:d027/64
enp4s0           DOWN
ian@attic4-u18:~$ ip -br -4  a
lo               UNKNOWN        127.0.0.1/8
enp2s0           UP             192.168.1.24/24

So far you have seen local loopback and Ethernet links. There are several other link types you may see, including wifi and bridge links. Listing 10 shows a brief IPv4 output for a Fedora system with a loopback address, an Ethernet link, a wifi link, and a bridge link for the benefit of possible virtual machines on my system.

[ian@ianattic5-f31 ~]$ ip -4 -br a
lo               UNKNOWN        127.0.0.1/8
enp9s0           UP             192.168.1.25/24
virbr0           DOWN           192.168.122.1/24
wlp8s0u2         UP             192.168.3.21/24

Another object you need to get traffic in and out of your system is a route. Use ip with the route object to display routes. If you do not see any v6 information, run the command again with the -6 option as shown in Listing 11.

Listing 11. Displaying route information using ip route
ian@attic4-u18:~$ ip route
default via 192.168.1.1 dev enp2s0 proto dhcp metric 103
169.254.0.0/16 dev enp2s0 scope link metric 1000
192.168.1.0/24 dev enp2s0 proto kernel scope link src 192.168.1.24 metric 103
ian@attic4-u18:~$ ip -6 route
2606:a000:1126:c8a8::/64 dev enp2s0 proto ra metric 103 pref medium
fe80::/64 dev enp2s0 proto kernel metric 103 pref medium
fe80::/64 dev enp2s0 proto kernel metric 256 pref medium
default via fe80::bc87:36ff:fe3f:c4b2 dev enp2s0 proto ra metric 103 pref medium

Looking at the first line of output you see a default IPv4 route, which will be used for all version 4 traffic unless a more specific route is found. Traffic will be sent and received over the Ethernet link, enp2s0. The proto field indicates that this route was set up when the address was assigned from a DHCP server. Finally, the metric is a measure that is used to give one route preference over alternate routes. The lower the metric the more the route is preferred.

You also see an entry for the IPv4 subnet 192.168.1.0/24. This also uses the Ethernet link enp2s0. The route was added by the kernel during initialization, and traffic sent over this route will have a source IP address of this host, namely 192.168.1.24.

This example was a fairly simple one with one active network link. On my Fedora system with both an Ethernet and a wifi link active, the route information is a little more complex. The version 4 information is shown in Listing 12.

[ian@ianattic5-f31 ~]$ ip route
default via 192.168.1.1 dev ) proto dhcp metric 100
default via 192.168.3.1 dev wlp8s0u2 proto dhcp metric 600
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.25 metric 100
192.168.3.0/24 dev wlp8s0u2 proto kernel scope link src 192.168.3.21 metric 600
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
[ian@ianattic5-f31 ~]$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2606:a000:1126:c8a8::/64 dev enp9s0 proto ra metric 100 pref medium
fe80::/64 dev enp9s0 proto kernel metric 100 pref medium
fe80::/64 dev wlp8s0u2 proto kernel metric 600 pref medium
default via fe80::bc87:36ff:fe3f:c4b2 dev enp9s0 proto ra metric 100 pref medium

In this case, there are default routes over both Ethernet (enp9s0) and wifi (wlp8s0u2). The Ethernet link has a metric of 100 while the wifi link has a higher metric of 600. So, the Ethernet link is preferred if nothing else overrides the routing decision. The wifi link is on a guest network (subnet 192.168.3.0/24) that has limited access to resources on the main network (subnet 192.168.3.0/24). Traffic destined for the 192.168.3.0/24 subnet will be sent over the wifi adapter with a source address of 192.168.3.21 while traffic for the 192.168.3.0/24 subnet as well as less specific traffic will use the Ethernet adapter and have the source address as 192.168.1.25.

Given the above routing information, you might wonder why traffic from 192.168.1.25 to itself would appear to be routed out through the Ethernet adapter rather than just being looped inside the host. And a similar question for 192.168.3.21 and the wifi adapter. And how is traffic to 127.0.0.1 routed? The answer is that the kernel maintains a local routing table for high-priority routing of broadcast and loopback traffic. Use the ip get command to see more details about a particular route. Add oif for output interface if you want to check the route over a specific link. Several illustrations are in Listing 13.

Listing 13. Using ip with the get commands
[ian@ianattic5-f31 ~]$ ip route get 127.0.0.1
local 127.0.0.1 dev lo src 127.0.0.1 uid 1000
    cache <local>
[ian@ianattic5-f31 ~]$ ip route get 192.168.1.25
local 192.168.1.25 dev lo src 192.168.1.25 uid 1000
    cache <local>
[ian@ianattic5-f31 ~]$ ip route get 192.168.3.21
local 192.168.3.21 dev lo src 192.168.3.21 uid 1000
    cache <local>
[ian@ianattic5-f31 ~]$ ip route get 8.8.8.8
8.8.8.8 via 192.168.1.1 dev enp9s0 src 192.168.1.25 uid 1000
    cache
[ian@ianattic5-f31 ~]$ ip route get 192.168.3.29
192.168.3.29 dev wlp8s0u2 src 192.168.3.21 uid 1000
    cache
[ian@ianattic5-f31 ~]$ ip route get 8.8.8.8 oif wlp8s0u2
8.8.8.8 via 192.168.3.1 dev wlp8s0u2 src 192.168.3.21 uid 1000
    cache

This quick introduction to displaying link, address, and route information does not cover all of the many options of the ip command that are available. Refer back to the list of more detailed man pages that I listed in the Detailed man pages for the ip command section.

Activate or deactivate interfaces using the ip command

Before you start configuring interfaces, it’s handy to know how to activate or deactivate them. Making changes to an already active interface is generally not a good idea! Listing 14 shows how to deactivate an interface and then reactivate it using the ip command to set the interface up or down. You need administrative authority to change the status of devices such as links.

Listing 14. Deactivating and activating an interface using ip
[ian@ianattic5-f31 l-lpic1-109-3]$ sudo ip link set enp9s0 down
[ian@ianattic5-f31 l-lpic1-109-3]$ ip -br addr show dev enp9s0
enp9s0           DOWN
[ian@ianattic5-f31 l-lpic1-109-3]$ sudo ip link set enp9s0 up
[ian@ianattic5-f31 l-lpic1-109-3]$ ip -br addr show dev enp9s0
enp9s0           UP             192.168.1.25/24 fe80::efaa:4539:a6ba:27b9/64

Saving and restoring interfaces and routes

It is also a good idea to know how to save and possibly restore the state of your network and routes. The ip command can save current information to a binary file using the save option. Use showdump to show what is in such a binary file and restore to restore it. Needless to say, you will need administrator privilege for restore operations. Listing 15 shows how to dump information for an Ethernet link and for the route information, and how to show the contents of the saved binary files.

Listing 15. Saving and restoring using ip
[ian@ianattic5-f31 l-lpic1-109-3]$ ip addr save enp9s0 >saved-enp9s0
[ian@ianattic5-f31 l-lpic1-109-3]$ ip addr showdump < saved-enp9s0
if2:
    inet 192.168.1.25/24 brd 192.168.1.255 scope global dynamic noprefixroute enp9s0
       valid_lft 73568sec preferred_lft 73568sec
if2:
    inet6 2606:a000:1126:c8a8:5a1e:d3c4:ffe:3894/64 scope global dynamic noprefixroute
       valid_lft 86382sec preferred_lft 86382sec
if2:
    inet6 fe80::efaa:4539:a6ba:27b9/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[ian@ianattic5-f31 l-lpic1-109-3]$ ip route save >saved-route
[ian@ianattic5-f31 l-lpic1-109-3]$ ip route showdump < saved-route
default via 192.168.1.1 dev enp9s0 proto dhcp metric 100
default via 192.168.3.1 dev wlp8s0u2 proto dhcp metric 600
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.25 metric 100
192.168.3.0/24 dev wlp8s0u2 proto kernel scope link src 192.168.3.21 metric 600
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Configure network interfaces and routes

You can use the ip command to configure links, addresses, or routes. Changes made using the ip command are not persistent. So, you will need to make the changes after each boot or else use persistent network configuration tools as described in our tutorial, “Learn Linux 101: Persistent network configuration“. Using ip is a great way to experiment and check out your planned changes.

The ip command has a limited amount of configuration ability with physical links such as Ethernet or wifi links. You can alter flags such as multicast for example and you can add or delete IP addresses. However, you can use ip to configure a large variety of virtual links, such as bridges, VLANs, or channel bonds. You can use ip link help to find the various link types, and add a type to find the parameters applicable to a particular link type. Listing 16 shows the end of the output for ip link help and the output for ip help bond.

ian@attic4-u18:~$ ip l help
Usage: ip link add [link DEV] [ name ] NAME
                   [ txqueuelen PACKETS ]
                   [ address LLADDR ]
                   [ broadcast LLADDR ]
                   [ mtu MTU ] [index IDX ]
                   [ numtxqueues QUEUE_COUNT ]
                   [ numrxqueues QUEUE_COUNT ]
                   type TYPE [ ARGS ]

...

       ip link help [ TYPE ]

TYPE := { vlan | veth | vcan | vxcan | dummy | ifb | macvlan | macvtap |
          bridge | bond | team | ipoib | ip6tnl | ipip | sit | vxlan |
          gre | gretap | erspan | ip6gre | ip6gretap | ip6erspan |
          vti | nlmon | team_slave | bond_slave | ipvlan | geneve |
          bridge_slave | vrf | macsec }
ian@attic4-u18:~$ ip link help bond
Usage: ... bond [ mode BONDMODE ] [ active_slave SLAVE_DEV ]
                [ clear_active_slave ] [ miimon MIIMON ]
                [ updelay UPDELAY ] [ downdelay DOWNDELAY ]
                [ use_carrier USE_CARRIER ]
                [ arp_interval ARP_INTERVAL ]
                [ arp_validate ARP_VALIDATE ]
                [ arp_all_targets ARP_ALL_TARGETS ]
                [ arp_ip_target [ ARP_IP_TARGET, ... ] ]
                [ primary SLAVE_DEV ]
                [ primary_reselect PRIMARY_RESELECT ]
                [ fail_over_mac FAIL_OVER_MAC ]
                [ xmit_hash_policy XMIT_HASH_POLICY ]
                [ resend_igmp RESEND_IGMP ]
                [ num_grat_arp|num_unsol_na NUM_GRAT_ARP|NUM_UNSOL_NA ]
                [ all_slaves_active ALL_SLAVES_ACTIVE ]
                [ min_links MIN_LINKS ]
                [ lp_interval LP_INTERVAL ]
                [ packets_per_slave PACKETS_PER_SLAVE ]
                [ tlb_dynamic_lb TLB_DYNAMIC_LB ]
                [ lacp_rate LACP_RATE ]
                [ ad_select AD_SELECT ]
                [ ad_user_port_key PORTKEY ]
                [ ad_actor_sys_prio SYSPRIO ]
                [ ad_actor_system LLADDR ]


BONDMODE := balance-rr|active-backup|balance-xor|broadcast|802.3ad|balance-tlb|balance-alb
ARP_VALIDATE := none|active|backup|all
ARP_ALL_TARGETS := any|all
PRIMARY_RESELECT := always|better|failure
FAIL_OVER_MAC := none|active|follow
XMIT_HASH_POLICY := layer2|layer2+3|layer3+4|encap2+3|encap3+4
LACP_RATE := slow|fast
AD_SELECT := stable|bandwidth|count

Now I will show you how to add an IP address to an existing link and how to update the routing table. For simplicity, I will restrict the discussion to IPv4 addresses, but the same general techniques work for IPv6 as well.

Recall that my Fedora system has a wifi adapter with an associated IP address of 192.168.3.21. The IPv4 address and route information is shown in Listing 17.

[[ian@ianattic5-f31 ~]$ ip -4 a show wlp8s0u2
3: wlp8s0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    inet 192.168.3.21/24 brd 192.168.3.255 scope global dynamic noprefixroute wlp8s0u2
       valid_lft 53458sec preferred_lft 53458sec
[ian@ianattic5-f31 ~]$ ip -4 r
default via 192.168.1.1 dev enp9s0 proto dhcp metric 100
default via 192.168.3.1 dev wlp8s0u2 proto dhcp metric 600
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.25 metric 100
192.168.3.0/24 dev wlp8s0u2 proto kernel scope link src 192.168.3.21 metric 600
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Suppose I want to change the address from 192.168.3.21 to 192.168.3.30 and also change the metric to 550. It is not possible to change the metric using the IP command. So, one way to do this is to first add the new address and new metric and then delete the original address.

First, I’ll add an address and then see what I accomplished as shown in Listing 18. Note that you need administration authority to add an address or do several of the other things I will do.

Listing 18. Using ip to add an IP address
[ian@ianattic5-f31 ~]$ sudo ip addr add 192.168.3.30/24 dev wlp8s0u2
[ian@ianattic5-f31 ~]$ ip -4 a show wlp8s0u2
3: wlp8s0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    inet 192.168.3.21/24 brd 192.168.3.255 scope global dynamic noprefixroute wlp8s0u2
       valid_lft 53338sec preferred_lft 53338sec
    inet 192.168.3.30/24 scope global secondary wlp8s0u2
       valid_lft forever preferred_lft forever

So far, so good, but I forgot to add a broadcast address and a metric. Listing 19 shows how to delete the address that I just added and add it again with a broadcast address and a metric. Note that brd is an adequate abbreviation for broadcast and the + tells ip to derive the broadcast address from the IP address by setting or resetting the host bits of the interface prefix. You can use either + or - for this purpose or you can spell out the broadcast address in full dotted-quad format.

Listing 19. Delete an address and add a broadcast addresses using ip
[ian@ianattic5-f31 ~]$ sudo ip addr del 192.168.3.30/24  dev wlp8s0u2
[ian@ianattic5-f31 ~]$ sudo ip addr add 192.168.3.30/24 brd + metric 550 dev wlp8s0u2
[ian@ianattic5-f31 ~]$ ip -4 a show wlp8s0u2
3: wlp8s0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    inet 192.168.3.21/24 brd 192.168.3.255 scope global dynamic noprefixroute wlp8s0u2
       valid_lft 50563sec preferred_lft 50563sec
    inet 192.168.3.30/24 metric 550 brd 192.168.3.255 scope global secondary wlp8s0u2
       valid_lft forever preferred_lft forever

Note that the new address has been added as a secondary address. There is no way to swap a secondary address for a primary address. Depending on the settings of certain sysctl variables secondary IPv4 addresses may be deleted or promoted when the primary address is deleted. This is not a problem for IPv5 addresses where secondaries are promoted if the primary is deleted. The default values may vary by Linux distribution. On the Ubuntu system, I am using the sysctl value related to promoting secondaries are shown in Listing 20.

Listing 20. Sysctl variables for promoting secondary IPv5 addresses
ian@attic4-u18:~$ sudo sysctl -a --pattern "net.ipv4.*_secondaries"
net.ipv4.conf.all.promote_secondaries = 1
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.enp2s0.promote_secondaries = 0
net.ipv4.conf.enp4s0.promote_secondaries = 0
net.ipv4.conf.lo.promote_secondaries = 0

Before I delete the original 192.168.3.21 address, I’ll check the existing routes to help me configure the necessary routes for the new address correctly. Two of my IPv4 route entries involve the wlp8s0u2 wifi adapter as shown in Listing 21. So, I will need routes like these each with metric 550 after I delete the 192.168.3.21 address.

Listing 21. IPv4 routes
[ian@ianattic5-f31 ~]$ ip -4 route
default via 192.168.1.1 dev enp9s0 proto dhcp metric 100
default via 192.168.3.1 dev wlp8s0u2 proto dhcp metric 600
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.25 metric 100
192.168.3.0/24 dev wlp8s0u2 proto kernel scope link src 192.168.3.21 metric 600
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Now I will delete 192.168.3.21 and show you how the address I added is promoted to secondary. In Listing 22, you see that the new address was promoted to primary and a link scope route was automatically added with metric 550, derived from the value I specified when I added the address.

Listing 22. Deleting the primary IPv4 addresses
[ian@ianattic5-f31 ~]$ sudo ip addr del 192.168.3.21/24 dev wlp8s0u2[ian@ianattic5-f31 ~]$ ip -4 addr show wlp8s0u2
3: wlp8s0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    inet 192.168.3.30/24 metric 550 brd 192.168.3.255 scope global wlp8s0u2
       valid_lft forever preferred_lft forever
[ian@ianattic5-f31 ~]$ ip -4 route
default via 192.168.1.1 dev enp9s0 proto dhcp metric 100
default via 192.168.3.1 dev wlp8s0u2 proto dhcp metric 600
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.25 metric 100
192.168.3.0/24 dev wlp8s0u2 proto kernel scope link src 192.168.3.30 metric 550
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

These changes do not update the default route over the wlp8s0u2 which still has the original metric of 600. You cannot change the metric on a route using the ip command. You have to delete the route and add it again. The addition does not inherit the metric value from the address so you need to specify it explicitly as shown in Listing 23.

Listing 23. Updating the default route
[ian@ianattic5-f31 ~]$ sudo ip route del default via 192.168.3.1
[ian@ianattic5-f31 ~]$ sudo ip route add default via 192.168.3.1 metric 550
[ian@ianattic5-f31 ~]$ ip -4 route
default via 192.168.1.1 dev enp9s0 proto dhcp metric 100
default via 192.168.3.1 dev wlp8s0u2 metric 550
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.25 metric 100
192.168.3.0/24 dev wlp8s0u2 proto kernel scope link src 192.168.3.30 metric 550
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

So now I have accomplished the changes I set out to illustrate. Remember that changes made using the ip command are transient and will be lost at the next reboot.

Debugging network configuration problems

This section explains how to debug network configuration problems.

Hostname

You may want to verify your hostname value as part of your debugging. While the hostname command is listed in the LPI objectives for this tutorial I’ll simply refer you to our tutorial “Learn Linux 101: Persistent network configuration“, where it and the associated hostnamectl command are both covered.

I already showed you how to use the ip command to display the link status. Assuming the status shows that the link is up, you might need to verify that you have connectivity over the link. My first check usually involves the ping command which sends an ICMP ECHO request to another host, or even to your own host. Use either the -4 or -6 option to force using either IPv4 or IPv5 if you are looking to check a particular connection type. On most systems, ping6 is an alternate command equivalent to ping -6. Use the -c option to limit the number of echo requests that are sent, otherwise ping will continue transmitting packets until you terminate it using Ctrl+C. Recall that the normal kernel routing will send traffic destined to a host address on your own system will use the local loopback link. Listing 24 shows some examples of this.

[ian@ianattic5-f31 ~]$ ping -c2 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.070 ms

--- 127.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1060ms
rtt min/avg/max/mdev = 0.070/0.099/0.129/0.029 ms
[ian@ianattic5-f31 ~]$ ping -c2 ::1
PING ::1(::1) 56 data bytes
64 bytes from ::1: icmp_seq=1 ttl=64 time=0.090 ms
64 bytes from ::1: icmp_seq=2 ttl=64 time=0.074 ms

--- ::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.074/0.082/0.090/0.008 ms
[ian@ianattic5-f31 ~]$ ping -c2 -4 192.168.3.30
PING 192.168.3.30 (192.168.3.30) 56(84) bytes of data.
64 bytes from 192.168.3.30: icmp_seq=1 ttl=64 time=0.125 ms
64 bytes from 192.168.3.30: icmp_seq=2 ttl=64 time=0.076 ms

--- 192.168.3.30 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.076/0.100/0.125/0.024 ms
[ian@ianattic5-f31 ~]$ ping -c2 -6 2606:a000:1126:c8a2:4dc7:4faf:f930:f31c
PING 2606:a000:1126:c8a2:4dc7:4faf:f930:f31c(2606:a000:1126:c8a2:4dc7:4faf:f930:f31c) 56 data bytes
64 bytes from 2606:a000:1126:c8a2:4dc7:4faf:f930:f31c: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 2606:a000:1126:c8a2:4dc7:4faf:f930:f31c: icmp_seq=2 ttl=64 time=0.089 ms

--- 2606:a000:1126:c8a2:4dc7:4faf:f930:f31c ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.089/0.103/0.118/0.014 ms

Other, perhaps more common, uses for ping are to check connectivity to a gateway, to another device on your corporate or local network, to a public DNS server or to an arbitrary host on the Internet. In general, start with the local interface and then move further away. Listing 25 shows how to ping my IPv4 gateway, a neighbor on my local network which has both version 4 and version 6 entries in my /etc/hosts file, and the Google public DNS server (8.8.8.8). I have used the -q (for quiet) option to just produce summary output.

Listing 25. Reaching out with ping
[ian@ianattic5-f31 ~]$ ping -q -c2 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.

--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1046ms
rtt min/avg/max/mdev = 0.391/0.421/0.452/0.030 ms
[ian@ianattic5-f31 ~]$ ping -4 -q -c2 attic4
PING attic4 (192.168.1.24) 56(84) bytes of data.

--- attic4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1014ms
rtt min/avg/max/mdev = 0.137/0.142/0.148/0.005 ms
[ian@ianattic5-f31 ~]$ ping -6 -q -c2 attic4
PING attic4(attic4 (2606:a000:1126:c8a2:e8c3:113c:6bbd:3647)) 56 data bytes

--- attic4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1062ms
rtt min/avg/max/mdev = 0.159/0.159/0.160/0.000 ms
[ian@ianattic5-f31 ~]$ ping -q -c2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 21.845/25.809/29.773/3.964 ms

Looking at these examples, it is not evident which interface was used to transmit and receive the ping data. Listing 26 shows how to use the -I (for interface) option with ping to force use of my wifi interface (wlp8s0u2). Note how the attempt to ping attic4 without specifying -4 or -6 results in a message saying “Network is unreachable”. In this case, it means that my wifi adapter is not set up to handle version 6 traffic and that’s what ping was trying to use. If you see this, you may want to add -4 or -6 to check whether you may have one or the other kind of connectivity only.

Listing 26. Pinging over a specific interface
[ian@ianattic5-f31 ~]$ ping -I -q -c2 wlp8s0u2 192.168.1.1
ping: wlp8s0u2: Name or service not known
[ian@ianattic5-f31 ~]$ ping -q -c2 -I wlp8s0u2 192.168.1.1
PING 192.168.1.1 (192.168.1.1) from 192.168.3.30 wlp8s0u2: 56(84) bytes of data.

--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 678.964/1182.229/1685.495/503.265 ms, pipe 2
[ian@ianattic5-f31 ~]$ ping -q -c2 -I wlp8s0u2 192.168.3.1
PING 192.168.3.1 (192.168.3.1) from 192.168.3.30 wlp8s0u2: 56(84) bytes of data.

--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 3.589/28.840/54.091/25.251 ms
[ian@ianattic5-f31 ~]$ ping -q -c2 -I wlp8s0u2 attic4
ping: connect: Network is unreachable
[ian@ianattic5-f31 ~]$ ping -4 -q -c2 -I wlp8s0u2 attic4
PING attic4 (192.168.1.24) from 192.168.3.30 wlp8s0u2: 56(84) bytes of data.

--- attic4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 1.542/460.310/919.078/458.768 ms

There are a number of other options that you can use to control the way ping works. Be aware that the options that you select may affect your ability to get a response.

Routes and paths

If you can’t reach a host using ping, you might try either traceroute or tracepath. Both are designed to probe a possible path to a destination. They have some similarities, but tracepath is newer and designed to not require privileged authority and also tracepath does not have as many options. Note that not all traceroute options require privilege, but some do. As with ping, each has a -4 or -6 option for IPv4 or IPv6. Some systems also have traceroute6 and tracepath6 with the obvious meanings. You can specify an interface with traceroute using the -i (not lower case as compared to ping) option, but this is not possible with tracepath. Both commands have a -m option to limit the number of hops that will be tried. By default, tracepath does not show IP addresses, use the -n option for just numeric addresses or the -b option to see both. Listing 27 shows some basic usage of tracepath.

Listing 27. Using tracepath to nearby nodes
[ian@ianattic5-f31 ~]$ tracepath 192.168.3.1
 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway                                             69.192ms reached
 1:  _gateway                                              1.847ms reached
     Resume: pmtu 1500 hops 1 back 1
[ian@ianattic5-f31 ~]$ tracepath -b attic4
 1?: [LOCALHOST]                        0.008ms pmtu 1500
 1:  attic4 (2606:a000:1126:c8a2:e8c3:113c:6bbd:3647)      0.241ms reached
 1:  attic4 (2606:a000:1126:c8a2:e8c3:113c:6bbd:3647)      0.144ms reached
     Resume: pmtu 1500 hops 1 back 1
[ian@ianattic5-f31 ~]$ tracepath -4 -n attic4
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.1.24                                          0.251ms reached
 1:  192.168.1.24                                          0.124ms reached
     Resume: pmtu 1500 hops 1 back 1

Listing 28 shows similar information from tracepath. In this case, I also use the -i option to check the path over the wifi adapter rather than the default Ethernet adapter.

Listing 28. Using traceroute to nearby nodes
[ian@ianattic5-f31 ~]$
[ian@ianattic5-f31 ~]$ traceroute 192.168.3.1
traceroute to 192.168.3.1 (192.168.3.1), 30 hops max, 60 byte packets
 1  _gateway (192.168.3.1)  2.348 ms  2.241 ms  2.500 ms
[ian@ianattic5-f31 ~]$ # Traffic via the WiFi gateway to the Ethernet host
[ian@ianattic5-f31 ~]$ sudo traceroute -i wlp8s0u2 192.168.1.24
traceroute to 192.168.1.24 (192.168.1.24), 30 hops max, 60 byte packets
 1  _gateway (192.168.3.1)  1411.191 ms  1411.195 ms  1411.163 ms
 2  attic4 (192.168.1.24)  1411.644 ms  1411.644 ms  1411.613 ms
[ian@ianattic5-f31 ~]$ traceroute attic4
traceroute to attic4 (192.168.1.24), 30 hops max, 60 byte packets
 1  attic4 (192.168.1.24)  1.078 ms  0.988 ms  0.857 ms
[ian@ianattic5-f31 ~]$ traceroute6 attic4
traceroute to attic4 (2606:a000:1126:c8a2:e8c3:113c:6bbd:3647), 30 hops max, 80 byte packets
 1  attic4 (2606:a000:1126:c8a2:e8c3:113c:6bbd:3647)  1.163 ms  1.061 ms  0.945 ms

When you try to probe further, either of these commands may fail to find a path. Use the -M option of traceroute to try different methods. Traditionally these are udp (default) or icmp (ICMP). Because either or both of these may be blocked by firewalls, a newer option, tcp, is available on some implementations. You can also use -I instead of -M icmp , and -T instead of -M tcp.

Listing 29 shows a comparison of tracepath and traceroute options to find a path from my system to lpi.org. In this case, only tracepath with the -T option succeeded.

Listing 29. Comparing tracepath and traceroute
[ian@ianattic5-f31 ~]$ tracepath -b -m 14 lpi.org
 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway (192.168.1.1)                                0.438ms
 1:  _gateway (192.168.1.1)                                0.394ms
 2:  no reply
 3:  no reply
 4:  cpe-024-025-041-002.ec.res.rr.com (24.25.41.2)       16.947ms
 5:  be31.chrcnctr01r.southeast.rr.com (24.93.64.186)     20.365ms
 6:  bu-ether11.atlngamq46w-bcr00.tbone.rr.com (66.109.6.34)  20.988ms
 7:  66.109.5.125 (66.109.5.125)                          18.059ms asymm  8
 8:  ae-10.edge4.Atlanta2.Level3.net (4.68.37.73)         35.395ms asymm  9
 9:  ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242)      45.442ms asymm 14
10:  4.28.138.82 (4.28.138.82)                            39.924ms asymm 13
11:  no reply
12:  no reply
13:  no reply
14:  no reply
     Too many hops: pmtu 1500
     Resume: pmtu 1500
[ian@ianattic5-f31 ~]$ sudo traceroute -I -m 14 lpi.org
traceroute to lpi.org (65.39.134.165), 14 hops max, 60 byte packets
 1  _gateway (192.168.1.1)  0.588 ms  0.750 ms  0.897 ms
 2  mta-107-13-224-1.nc.rr.com (107.13.224.1)  13.896 ms  13.946 ms  13.993 ms
 3  cpe-174-111-118-177.triad.res.rr.com (174.111.118.177)  32.620 ms  32.610 ms  32.651 ms
 4  cpe-024-025-041-002.ec.res.rr.com (24.25.41.2)  24.789 ms  24.814 ms  24.854 ms
 5  be31.chrcnctr01r.southeast.rr.com (24.93.64.186)  31.228 ms  31.220 ms  31.263 ms
 6  bu-ether14.atlngamq46w-bcr00.tbone.rr.com (66.109.6.82)  34.075 ms  22.140 ms  28.131 ms
 7  66.109.5.125 (66.109.5.125)  20.755 ms  33.047 ms  33.114 ms
 8  ae-10.edge4.Atlanta2.Level3.net (4.68.37.73)  33.210 ms  33.145 ms  33.240 ms
 9  ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242)  49.789 ms  49.812 ms  49.848 ms
10  4.28.138.82 (4.28.138.82)  51.225 ms  51.249 ms  51.272 ms
11  * * *
12  * * *
13  * * *
14  * * *
[ian@ianattic5-f31 ~]$ sudo traceroute -T -m 14 lpi.org
traceroute to lpi.org (65.39.134.165), 14 hops max, 60 byte packets
 1  _gateway (192.168.1.1)  0.363 ms  0.291 ms  0.512 ms
 2  * * *
 3  * * *
 4  cpe-024-025-041-002.ec.res.rr.com (24.25.41.2)  20.366 ms  20.452 ms  20.348 ms
 5  be31.chrcnctr01r.southeast.rr.com (24.93.64.186)  27.943 ms  27.954 ms  30.316 ms
 6  bu-ether11.atlngamq46w-bcr00.tbone.rr.com (66.109.6.34)  36.744 ms  24.255 ms  32.746 ms
 7  66.109.5.125 (66.109.5.125)  42.821 ms  28.214 ms  28.328 ms
 8  ae-10.edge4.Atlanta2.Level3.net (4.68.37.73)  28.706 ms  21.630 ms  30.302 ms
 9  4.69.216.246 (4.69.216.246)  46.304 ms ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242)  
 39.646 ms 4.69.216.246 (4.69.216.246)  31.685 ms
10  4.28.138.82 (4.28.138.82)  34.332 ms  44.060 ms  43.585 ms
11  * * *
12  65.39.134.165 (65.39.134.165)  44.264 ms  31.267 ms  39.192 ms

You can find more about other available options for both these commands in the man or info pages.

As you saw above, it can be challenging to verify a path from your host to an arbitrary Internet host. Usually these tools work well in parts of the network under your or your organization’s control. You also saw that using Transmission Control Protocol (TCP) for probes did work in one case where both User Datagram Protocol (UDP) and Internet Control Message Protocol (ICMP) failed. Using TCP effectively attempts to start a conversation that is not typically blocked, such as a browser or perhaps a Secure Shell (SSH) session. If there is no listener on the port, the session is rejected by the target host. If there is a listener, your host receives a positive session-setup response and immediately cancels the session before normal data transfer begins.

Sockets and ss

The ss command is part of the iproute2 package. Use it to dump socket statistics. If used without parameters, it dumps a list of open non-listening sockets (for example, TCP/UNIX/UDP) that have established connections. This can be quite long. You can use grep for simple filtering, in which case it is useful to add the -o (one line) option to get multiline output on a single line. Use the -t or -u options for TCP or UDP sockets respectively and the -l option for listening sockets. The -s option shows you a short summary of sockets. Listing 30 shows a few examples.

Listing 30. Using the ss command
[ian@ianattic5-f31 ~]$ ss -s
Total: 1210
TCP:   29 (estab 6, closed 13, orphaned 0, timewait 0)

Transport Total     IP        IPv6
RAW         2         0         2
UDP        10         7         3
TCP        16        12         4
INET       28        19         9
FRAG        0         0         0

[ian@ianattic5-f31 ~]$ ss -ul
State   Recv-Q  Send-Q         Local Address:Port     Peer Address:Port Process
UNCONN  0       0                    0.0.0.0:37013         0.0.0.0:*
UNCONN  0       0                    0.0.0.0:mdns          0.0.0.0:*
UNCONN  0       0              192.168.122.1:domain        0.0.0.0:*
UNCONN  0       0             0.0.0.0%virbr0:bootps        0.0.0.0:*
UNCONN  0       0        192.168.1.25%enp9s0:bootpc        0.0.0.0:*
UNCONN  0       0                    0.0.0.0:xdmcp         0.0.0.0:*
UNCONN  0       0                  127.0.0.1:323           0.0.0.0:*
UNCONN  0       0                       [::]:mdns             [::]:*
UNCONN  0       0                      [::1]:323              [::]:*
UNCONN  0       0                       [::]:52184            [::]:*

[ian@ianattic5-f31 ~]$ ss -o | grep http
tcp               ESTAB                  0                   0
192.168.1.25:60764                                        35.161.207.109:https
timer:(keepalive,4min2sec,0)
tcp               ESTAB                  0                   0
192.168.1.25:54274                                       192.241.243.150:https
timer:(keepalive,2min14sec,0)
tcp               ESTAB                  0                   0
192.168.1.25:40932                                        172.217.15.110:https
tcp               CLOSE-WAIT             1                   0
192.168.1.25:33990                                          99.84.221.57:https
tcp               ESTAB                  0                   0
192.168.1.25:39522                               172.217.5.234:https

The ss command also has the ability to construct quite elaborate state filters as part of the command. Listing 31 shows how to filter established TCP sockets using the SSH port, either as a source port or a destination port. It shows an example of four SSH sessions, two originating at my Ubuntu host (192.168.1.24) to my Fedora host (192.168.1.25 and 192.168.3.30) and two in the reverse direction, one V4 and one v6.

Listing 31. Using ss state filters
[ian@ianattic5-f31 ~]$ ss -t state established '( dport = :ssh or sport = :ssh )'
Recv-Q     Send-Q                              Local Address:Port
Peer Address:Port         Process
0          0                                    192.168.1.25:ssh
192.168.1.24:34946
0          0                                    192.168.1.25:55160
192.168.1.24:ssh
0          0                                    192.168.3.30:ssh
192.168.1.24:46028
0          0           [2606:a000:1126:c8a2:4dc7:4faf:f930:f31c]:ssh
[2606:a000:1126:c8a2:cd32:dd01:8c3f:434d]:52854

Note that the man page for ss says “Please take a look at the official documentation for details regarding filters.” Such details do not seem to exist in the official documentation packages as of the time of writing, although the examples in the man page are a very useful start.

Another socket tool that you can use is netcat. There are at least two versions and the name is sometimes nc or ncat. This is somewhat similar to the cat command in that you can use it to transmit text, command output, or files to another host. It operates in connect or listen mode. The listener does not have to be another netcat instance. In Listing 32, I have shown how to use ncat on my Fedora system to connect to the IBM web server on port 80. I typed in two lines to request the HTTP headers and you see the beginning of the response.

Listing 32. Using ncat (netcat) to retrieve HTTP headers
[ian@ianattic5-f31 ~]$ ncat -C www.ibm.com 80
GET /get HTTP/1.1
Host:www.ibm.com

HTTP/1.1 301 Moved Permanently
Server: AkamaiGHost
Content-Length: 0
Location: https://www.ibm.com/get
Date: Mon, 04 May 2020 15:36:20 GMT
Connection: keep-alive
...

You can find more information about the various modes and options in the man pages or in examples online.

Legacy net-tools commands

Before iproute2 there was net-tools. The net-tools package is still in use, a testament to its usefulness. Table 1 shows some of the commands I have used in this tutorial and the legacy command you could use instead. Use the man pages to explore other options of these commands.

Table 1. Comparison of new and legacy commands
New command Legacy command
ip a ifconfig -a
ip link set enp9s0 down ifconfig enp9s0 down
ip link set enp9s0 up ifconfig enp9s0 up
ip addr add 192.168.3.30/24 ifconfig add 192.168.3.30 dev wlp8s0u2
ifconfig wlp8s0u2 netmask 255.255.255.0
ip r route
ip route add default via 192.168.3.1 route add default gw 192.168.3.1
ip neigh arp -a
ss netstat

Conclusion

This concludes your introduction to Topic 109.3: Basic network troubleshooting.

Resources

Ian Shields