Cloned Ubuntu DHCP Issues

I just ran into an interesting issue with DHCP. I maintain a cluster of Ubuntu servers for a MooseFS cluster. MooseFS is pretty amazing, it is a clustered filesystem that allows simultaneous access to files from multiple clients. One of the great features of MooseFS is that it has great performance on commodity hardware. So it is relatively inexpensive to Set Up, and Manage. After using it for a while in our initial configuration we decided it was time to upgrade. Specifically to add some additional space in the form of three new chunk server nodes.

The beauty of MooseFS is that space, redundancy and performance can be added by simply adding more nodes. So to make things simple (Not that it is difficult) after we provisioned some new hardware. I simply cloned of one of the existing Chunk servers onto the new hardware. I didn’t realize that I would run into cloned Ubuntu DHCP issues.

Discovering the Cloned Ubuntu DHCP Issues

After I finished the cloning, I plugged in the new server on my work bench to finish up the config changes. Our server setup had a static address for the interfaces we have bonded to handle the MooseFS traffic. But I set the management interface to use DHCP.

The configuration process for the first server went quickly and I left it running while I started on server #2. As soon as server #2 booted up I realized that it had the same IP address as the first server. That sent me checking both servers for any static IP settings or other possible conflicts. But seeing no issues in the config I also checked our DHCP server and noticed that there were several leases to both servers with the same IP.

This got me digging around online for a bit for an answer. I have been using Linux for decades, so some of the newer systems/services keep me learning. In this case it was Netplan, and I installed it with Ubuntu on this machine. I found that Netplan does not use the interface MAC address as the default identifier for DHCP. It uses a generated DUID instead of the MAC address unless you specify otherwise.

I found the issue, as I cloned the chunk server I also cloned the DUID that NetPlan was using. So each clone looked the same to the DHCP server.

Fixing the DHCP Issues

Fortunately the fix for this problem is very simple. The DHCP entry needs to have:

dhcp-identifier: mac

With that line added to your Netplan setup it will default to the MAC address for the interface and your issue should be resolved.

Here is the full config with the new line in place for context. I hope it helps someone avoid the same issue with your own setup.

network:
  bonds:
    bond0:
      addresses:
      - 172.16.10.25/23
      interfaces:
      - ens1
      - ens1d1
      nameservers:
        addresses:
        - 172.16.10.2
        - 172.16.10.3
        search:
        - bdoga.com
      parameters:
        lacp-rate: fast
        mode: 802.3ad
        transmit-hash-policy: layer3+4
  ethernets:
    eno1: 
      dhcp4: yes
      dhcp-identifier: mac
    ens1: {}
    ens1d1: {}
  version: 2 

If that helped I also recommend this article I wrote about finding a replacement for Netstat.

Replacement for Netstat on Linux

Linux distributions grow, get updated, and more advanced. Most tools that you have grown accustomed to using remain, But a few have been replaced. Many reasons for the replacement of core tools. Most of these tools are removed because they are inefficient and replaced with better alternatives. Old networking tools like netstat and ifconfig work by accessing files in the /proc filesystem to aggregate their response. This behavior has been fine for years, especially on smaller systems. As systems grow and complexity increases the chances for inaccurate responses have increased. So what is the replacement for netstat?

The Linux networking subsystem has continued to evolve with the changing of the Linux Kernel, which has lead to new networking functions and processes. The new tools leverage Netlink Sockets to provide their information, which are much more efficient. For a full writeup on the reasons for the switch over I recommend this article on the real reasons of the Linux replacement for netstat.

Netstat is my goto tool for getting a list of open ports on a system and associated processes. Even with the changes in the works I have loaded the program from the net-tools package, until I couldn’t. Recently I was helping a client with a OpenSuse 15 system that didn’t have netstat, and it could not locate the package when I attempted to install it. I then tried the alternative to netstat, ss.

SS is the replacement for netstat

SS stands for “Socket Statistics” and operates in a manner similar to netstat. It formats things differently, leading to some required adjustments. Here is the basic help output for the ss command.

SS Usage

bdoga@webserver:~$ ss --help
 Usage: ss [ OPTIONS ]
        ss [ OPTIONS ] [ FILTER ]
    -h, --help          this message
    -V, --version       output version information
    -n, --numeric       don't resolve service names
    -r, --resolve       resolve host names
    -a, --all           display all sockets
    -l, --listening     display listening sockets
    -o, --options       show timer information
    -e, --extended      show detailed socket information
    -m, --memory        show socket memory usage
    -p, --processes     show process using socket
    -i, --info          show internal TCP information
    -s, --summary       show socket usage summary
    -b, --bpf           show bpf filter socket information
    -E, --events        continually display sockets as they are destroyed
    -Z, --context       display process SELinux security contexts
    -z, --contexts      display process and socket SELinux security contexts
    -N, --net           switch to the specified network namespace name
 

    -4, --ipv4          display only IP version 4 sockets
    -6, --ipv6          display only IP version 6 sockets
    -0, --packet        display PACKET sockets
    -t, --tcp           display only TCP sockets
    -u, --udp           display only UDP sockets
    -d, --dccp          display only DCCP sockets
    -w, --raw           display only RAW sockets
    -x, --unix          display only Unix domain sockets
    -f, --family=FAMILY display sockets of type FAMILY
 

    -A, --query=QUERY, --socket=QUERY
        QUERY := {all|inet|tcp|udp|raw|unix|unix_dgram|unix_stream|unix_seqpacket|packet|netlink}[,QUERY]
 

    -D, --diag=FILE     Dump raw information about TCP sockets to FILE
    -F, --filter=FILE   read filter information from FILE
        FILTER := [ state STATE-FILTER ] [ EXPRESSION ]
        STATE-FILTER := {all|connected|synchronized|bucket|big|TCP-STATES}
          TCP-STATES := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|closed|close-wait|last-ack|listen|closing}
           connected := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
        synchronized := {established|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
              bucket := {syn-recv|time-wait}
                 big := {established|syn-sent|fin-wait-{1,2}|closed|close-wait|last-ack|listen|closing} 

So while it may not be netstat it appears to be a very robust tool. Ss allows the querying of specific information and then filter the results directly. It is useful to filter directly without relying on a tool like grep or awk. But there is nothing to stop you from piping the output into the tool of your choice.

SS Usage Examples

bdoga@webserver:~$ ss -lt
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                    
 LISTEN     0      128        *:6032                     *:*                    
 LISTEN     0      128        *:http                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:ssh                      *:*                    
 LISTEN     0      128        *:https                    *:*                    
 LISTEN     0      128       :::ssh                     :::*               

Show all the tcp ports “-t” that the server is listening “-l” to.

bdoga@webserver:~$ sudo ss -ltp
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("proxysql_galera",pid=28271,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=950,fd=4)) 

Show all the tcp ports “-t” that the server is listening “-l” on. What process “-p” are using those ports. The “-p” flag requires root/sudo access to properly display which processes are using those ports.

bdoga@webserver:~$ sudo ss -lup
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 UNCONN     0      0          *:59879                    *:*                     users:(("rsyslogd",pid=556,fd=5))
 UNCONN     0      0      192.168.28.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=21))
 UNCONN     0      0      10.10.10.28:ntp                *:*                     users:(("ntpd",pid=9421,fd=20))
 UNCONN     0      0      209.33.221.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=19))
 UNCONN     0      0      127.0.0.1:ntp                  *:*                     users:(("ntpd",pid=9421,fd=18))
 UNCONN     0      0          *:ntp                      *:*                     users:(("ntpd",pid=9421,fd=17))
 UNCONN     0      0          *:snmp                     *:*                     users:(("snmpd",pid=1045,fd=7))
 UNCONN     0      0          *:https                    *:*                     users:(("litespeed",pid=21324,fd=18),("litespeed",pid=21321,fd=18))
 UNCONN     0      0      127.0.0.1:8822                 *:*                     users:(("Site24x7Agent",pid=1141,fd=19))
 UNCONN     0      0      fe80::218:3eff:fe57:7bf2%eth1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=24))
 UNCONN     0      0      fe80::216:3cff:fe4e:8673%eth0:ntp                     :::*                     users:(("ntpd",pid=9421,fd=23))
 UNCONN     0      0        ::1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=22))
 UNCONN     0      0         :::ntp                     :::*                     users:(("ntpd",pid=9421,fd=16)) 

Same as the previous example but listing udp ports “-u”.

bdoga@webserver:~$ sudo ss -at4p
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("mysql",pid=28336,fd=28),("timeout",pid=28335,fd=28),("proxysql_galera",pid=28333,fd=28),("proxysql_galera",pid=28332,fd=28),("proxysql_galera",pid=27839,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=22),("timeout",pid=28335,fd=22),("proxysql_galera",pid=28333,fd=22),("proxysql_galera",pid=28332,fd=22),("proxysql_galera",pid=27839,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=21),("timeout",pid=28335,fd=21),("proxysql_galera",pid=28333,fd=21),("proxysql_galera",pid=28332,fd=21),("proxysql_galera",pid=27839,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=20),("timeout",pid=28335,fd=20),("proxysql_galera",pid=28333,fd=20),("proxysql_galera",pid=28332,fd=20),("proxysql_galera",pid=27839,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=19),("timeout",pid=28335,fd=19),("proxysql_galera",pid=28333,fd=19),("proxysql_galera",pid=28332,fd=19),("proxysql_galera",pid=27839,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 TIME-WAIT  0      0      209.33.221.34:58184                209.33.221.83:mysql                
 TIME-WAIT  0      0      127.0.0.1:34804             127.0.0.1:6032                 
 TIME-WAIT  0      0      127.0.0.1:34470             127.0.0.1:6032                 
 TIME-WAIT  0      0      209.33.221.34:45368         209.33.221.36:mysql                
 ESTAB      0      0      209.33.221.34:37194         209.33.221.67:mysql                 users:(("mysql",pid=28336,fd=109),("timeout",pid=28335,fd=109),("proxysql_galera",pid=28333,fd=109),("proxysql_galera",pid=28332,fd=109),("proxysql_galera",pid=27839,fd=109),("proxysql",pid=3649,fd=109))
 TIME-WAIT  0      0      127.0.0.1:33812                127.0.0.1:6032                  

To show all “-a” tcp “-t” but only IPv4 “-4” interfaces and their associated processes “-p”.

I will take a little while to fully make the switch since I have used netstat for decades. Although it appears the future is bright, and ss should be able to take care of me in the future. Hopefully this overview helps you make the switch as well.

Here are some of the resources that I used when researching this topic:

Find Open Ports in Linux

Want to learn how to Force Disconnect iscsi and Reconnect iscsi on centos6+, check out this post to find out how.