Cloned Ubuntu DHCP Issues

I just ran into an interesting issue with DHCP. I maintain a cluster of Ubuntu servers for a MooseFS cluster. MooseFS is pretty amazing, it is a clustered filesystem that allows simultaneous access to files from multiple clients. One of the great features of MooseFS is that it has great performance on commodity hardware. So it is relatively inexpensive to Set Up, and Manage. After using it for a while in our initial configuration we decided it was time to upgrade. Specifically to add some additional space in the form of three new chunk server nodes.

The beauty of MooseFS is that space, redundancy and performance can be added by simply adding more nodes. So to make things simple (Not that it is difficult) after we provisioned some new hardware. I simply cloned of one of the existing Chunk servers onto the new hardware. I didn’t realize that I would run into cloned Ubuntu DHCP issues.

Discovering the Cloned Ubuntu DHCP Issues

After I finished the cloning, I plugged in the new server on my work bench to finish up the config changes. Our server setup had a static address for the interfaces we have bonded to handle the MooseFS traffic. But I set the management interface to use DHCP.

The configuration process for the first server went quickly and I left it running while I started on server #2. As soon as server #2 booted up I realized that it had the same IP address as the first server. That sent me checking both servers for any static IP settings or other possible conflicts. But seeing no issues in the config I also checked our DHCP server and noticed that there were several leases to both servers with the same IP.

This got me digging around online for a bit for an answer. I have been using Linux for decades, so some of the newer systems/services keep me learning. In this case it was Netplan, and I installed it with Ubuntu on this machine. I found that Netplan does not use the interface MAC address as the default identifier for DHCP. It uses a generated DUID instead of the MAC address unless you specify otherwise.

I found the issue, as I cloned the chunk server I also cloned the DUID that NetPlan was using. So each clone looked the same to the DHCP server.

Fixing the DHCP Issues

Fortunately the fix for this problem is very simple. The DHCP entry needs to have:

dhcp-identifier: mac

With that line added to your Netplan setup it will default to the MAC address for the interface and your issue should be resolved.

Here is the full config with the new line in place for context. I hope it helps someone avoid the same issue with your own setup.

network:
  bonds:
    bond0:
      addresses:
      - 172.16.10.25/23
      interfaces:
      - ens1
      - ens1d1
      nameservers:
        addresses:
        - 172.16.10.2
        - 172.16.10.3
        search:
        - bdoga.com
      parameters:
        lacp-rate: fast
        mode: 802.3ad
        transmit-hash-policy: layer3+4
  ethernets:
    eno1: 
      dhcp4: yes
      dhcp-identifier: mac
    ens1: {}
    ens1d1: {}
  version: 2 

If that helped I also recommend this article I wrote about finding a replacement for Netstat.

How To Run Program Before Login Prompt Ubuntu

I recently installed a new server in my home office. I typically just leave my servers to run headless. But with an old monitor laying around and plenty of idle CPU time I decided to play a bit. I mounted the monitor to my office rack and then started to work.

Rather than just display the normal text login prompt, I wanted it to show something cool at boot. I started to dig around on the web and found this article. It quickly described how to run a program before the login prompt on Ubuntu 16.04+.

Run Your Program before login

So I wrote a simple script /root/loginMatrix.sh which would simply run cmatrix on the main (tty1) console. Once I exited cmatrix it would display the normal login prompt. The sample script is as follows:

#!/bin/sh
/usr/bin/cmatrix -abs
exec /bin/login

I then edited the config file for getty@tty1 here (for Ubuntu 16.04+ only, not sure on other distrubutions):

/etc/systemd/system/getty@tty1.service.d/override.conf

I changed the contents to be:

[Service]
ExecStart=
ExecStart=-/root/loginMatrix.sh
StandardInput=tty
StandardOutput=tty

and then I ran the following command to activate it:

systemctl daemon-reload; systemctl restart getty@tty1.service

After the change the system started to show the cmatrix terminal animation immediately. But once I quit the application it was back to the login prompt.

how to run a program before the login prompt - Cmatrix running on the console prior to the login prompt
CMATRIX running on the console

Getting tricky

After running cmatrix for a few days straight, I decided that I wanted to change it up a bit. So I made a few adjustments to the /root/loginMatrix.sh script to make it a bit more dynamic. With the following changes I was now able to display something different each time I used the command prompt.

 #!/bin/bash
 declare -a arr=("/usr/bin/cmatrix -abs" "/snap/bin/asciiquarium" "/usr/sbin/iftop" "/usr/bin/htop")
 size=${#arr[@]}
 index=$(($RANDOM % $size))
 eval "${arr[$index]}"
 
 exec /bin/login 

These changes told the script to randomly choose either, cmatrix, asciiquarium, iftop, or htop and execute it. Then as before once I quit the application that was randomly executed it would again display the login prompt. My kids got way to excited when asciiquarium was chosen and had to watch the fish swim by. This solution worked for a while, but eventually I got tired of having to change the displayed program manually. So I started playing with options to automate the program change.

Automating the switch

These changes got a bit trickier. The script had to track the application PID so it could kill it when the timeout was reached. After trying several different methods I finally ran across this basic method for timing out a process. And the process isn’t perfect, but it does rotate through the different options on a ten minute interval. So that works, but the exiting to the login prompt doesn’t. So it’s only most of the way there. Here is my current /root/loginMatrix.sh script:

#!/bin/bash
declare -a arr=("/usr/bin/cmatrix -abs" "/snap/bin/asciiquarium" "/usr/sbin/iftop" "/usr/bin/cacafire")
size=${#arr[@]}
continue=1
timeout=600
interval=1
 
while [ $continue -eq 1 ]
do
index=$(($RANDOM % $size))
eval "${arr[$index]} &"
cmdpid=$!

 ((t = timeout))
 
     while ((t > 0)); do
         sleep 1
         kill -0 $cmdpid || exit 0
         ((t -= interval))
     done

 exit_status=$?
 echo $exit_status > ext.txt 
 if [[ $exit_status -ne 1 ]]; then
     continue=0
 fi
 
 kill -s SIGTERM $cmdpid && kill -0 $cmdpid || exit 0
 sleep 1
 #kill -s SIGKILL $cmdpid
 
 done
 
 exec /bin/login 

So this script accomplishes the switching of applications on the primary console. And I was able to add cacafire to the mix for a nice colored ascii fire animation. But if I have to use the console for the login, I will have to hit ctrl-alt-F2 and switch over to tty2. That won’t be the end of the world, lol. And in the meantime I have some fun console effects to keep my office interesting.

how to run a program before the login prompt - Cacafire running on the console prior to the login prompt
CACAFIRE running on the console

Did you like this article on how to run a program before the login prompt? If so you may like this article on how to change your hostname on Centos

Change Your Hostname in CentOS 8

Changing your computer or servers hostname is an infrequent activity for most. But if you are like me periodically I will hastily provision a VM. And only realize after the provisioning is complete that I should have used a more descriptive hostname. Or to have chosen a hostname that fits in the theme of the other servers (Middle Earth, Stormlight Archive, Planets, etc…). But sometimes that process can be tedious and end up with you questioning if you got it right. Fortunately it is easy to change your hostname in CentOS 8.

The ever useful “hostnamectl” command makes this a simple process. If you execute the command with no options it will give you the current hostname as well as many details about the system.

[bdoga@host ~]$ hostnamectl
   Static hostname: host.bdoga.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: b1ce9c049f6d4a9589ad540ae9aa1c43
           Boot ID: 1906ec0120c246aa84bd407e46a237b6
    Virtualization: kvm
  Operating System: CentOS Linux 8 (Core)
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.18.0-147.8.1.el8.lve.1.x86_64
      Architecture: x86-64

Change Your Hostname in CentOS 8

As shown in the example above, this servers hostname is “host.bdoga.local”. But I am ready for a change, and want to start naming my servers with Stormlight Archive Names. One of my favorite characters is Kaladin, and I want to have this server on my full domain “bdoga.com”. So to change the domain name to “Kaladin.bdoga.com” I would issue the following command.

[bdoga@host ~]$ sudo hostnamectl set-hostname kaladin.bdoga.com

After issuing the command you will not see any sort of confirmation. You should just be greeted with an empty command prompt, but with your new hostname.

[bdoga@host ~]$ sudo hostnamectl set-hostname kaladin.bdoga.com
[bdoga@kaladin ~]$

And there you have it, you have changed your hostname in CentOS 8. This method should also work for Ubuntu 16.04+, Debian 8.0+, CentOS 7+, and other Systemd based systems.

To learn some more details about this and other tools for changing your hostname on Centos 8 please visit linuxize’s post.

And feel free to check out some more of our content regarding CentOS based systems. Or visit some of our posts that will help you increase your Command Line prowess.

Fix Apt NO_PUBKEY Error

If you have used Debian, Ubuntu, Mint or any other linux distribution that uses APT based package management system. You are sure to have run into the NO_PUBKEY error. It can be marginally frustrating but fortunately it can be easy to fix the apt NO_PUBKEY error and get your system back up and ready to roll.

What is the NO_PUBKEY error?

The APT NO_PUBKEY error shows up when the public/private key pair has changed for one of your APT repositories. When this happens, if your local system or server does not have the correct public key, then it cannot verify the repository. And therefore you get the error. This process is in place to ensure you don’t accidentally download packages from an unknown APT source.

Fix the NO_PUBKEY error

There is a simple command that you can run to download the missing public key from one of the APT key servers. You will just need to replace the portion of the command that says “THE_MISSING_KEY_HERE” with the key that is reported in the error.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys THE_MISSING_KEY_HERE

So if you receive the following error

W: Failed to fetch http://ppa.launchpad.net/myrepository/apps/ubuntu/dists/bionic/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EA8CACC073C3DB2A

you would run the following command to get the working public key for the apt repository.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EA8CACC073C3DB2A

After the key has been updated you can then run your “apt update” and it should complete successfully.

Fix Multiple Keys with One Command

The following command can be used to fix multiple NO_PUBKEY errors with one command. Or can be used to fix a single NO_PUBKEY error without having to edit the command. It might be overkill but will still get the job done.

sudo apt update 2>&1 1>/dev/null | sed -ne 's/.*NO_PUBKEY //p' | while read key; do if ! [[ ${keys[*]} =~ "$key" ]]; then sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys "$key"; keys+=("$key"); fi; done

So now you know how to perform a Fix APT NO_PUBKEY error. This will keep you up and running, and ensure that you don’t fall behind on your package updates.

For additional details check out Linux Uprisings article about fixing NO_PUBKEY errors.

If you like this post, you might also like my post about how to Recursively Count the number of folders in a directory.

Change the SNMP Log Level in Ubuntu

The default SNMP settings for a Ubuntu server can end up filling your syslog file with tons of unnecessary entries. This makes it virtually impossible to sift through for anything which is actually useful. So it can be very advantageous to change the SNMP log level in Ubuntu.

I have a cacti setup which I use to log and report on the details of many linux and windows servers. This tool is amazing, and really gives me some great information to diagnose issues. Or catch issues as they are progressing, but before they become urgent. Sometimes it is just easier to see something when your data is represented visually.

Cacti relies upon SNMP as the technology to grab data from the machines or devices that it is monitoring. SNMP is an industry standard, supported by all major operating systems and network enabled devices. But by default, at least in Ubuntu, the log level is set so high that every SNMP request that comes to the server is reported in your syslog file. Cacti polls lots of different SNMP records to build its graphs. Under those default settings it can leave dozens of entries in the syslog every 5 minutes. As you could imagine this can quickly fill up your log file and make it virtually unusable. Fortunately we just need to make a quick adjustment in order to change the SNMP log level in Ubuntu. Here is a quick example of some of the Syslog entries that I you may be receiving.

Jul 8 06:28:48 server snmpd[7885]: error on subcontainer 'ia_addr' insert (-1)
Jul 8 06:29:18 server snmpd[7885]: error on subcontainer 'ia_addr' insert (-1)
Jul 8 06:29:48 server snmpd[7885]: error on subcontainer 'ia_addr' insert (-1)
Jul 8 06:30:02 server snmpd[7885]: Connection from UDP: [Originating IP]:41028->[Current Host IP]:161
Jul 8 06:30:02 server snmpd[7885]: Connection from UDP: [Originating IP]:48694->[Current Host IP]:161
Jul 8 06:30:02 server snmpd[7885]: Connection from UDP: [Originating IP]:39372->[Current Host IP]:161
Jul 8 06:30:02 server snmpd[7885]: Connection from UDP: [Originating IP]:54823->[Current Host IP]:161

Change the SNMP Log Level in Ubuntu

The change is just a quick flag in the /etc/default/snmpd file which changes how the system logs SNMP requests. The different log levels that are available are:

0 or ! for LOG_EMERG
1 or a for LOG_ALERT
2 or c for LOG_CRIT
3 or e for LOG_ERR
4 or w for LOG_WARNING
5 or n for LOG_NOTICE
6 or i for LOG_INFO
7 or d for LOG_DEBUG

By default a log level is not set so it is either dumping at the info or debug level. I prefer to switch it to level 3 (Error) which ensures that I still see any errors that come through. But doesn’t tell me every time a connection is made. This change can be made very easily. Basically you can just open up the /etc/default/snmpd file in your favorite editor and change the following line (Ubuntu 14.04 and 16.04).

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'

To look like this:

SNMPDOPTS='-LS3d -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'

The only part that changed was the “-Lsd” flags that changed to be “-LS3d”. The default entry is a little different between 14.04/16.04, 18.04 and 20.04. But I have included a few single commands you can copy/paste into your terminal to make the change.

Copy/Paste Command Line Changes

For Ubuntu 14.04 and 16.04:

sed -i -- "s@SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'@SNMPDOPTS='-LS3d -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'@g" /etc/default/snmpd
service snmpd restart

In Ubuntu 18.04:

sed -i -- "s@SNMPDOPTS='-Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'@SNMPDOPTS='-LS3d -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'@g" /etc/default/snmpd
service snmpd restart

Finally Ubuntu 20.04:

sed -i -- "s@#SNMPDOPTS='-LSwd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'@SNMPDOPTS='-LS3d -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'@g" /etc/default/snmpd
service snmpd restart

So there you go, now you can stop those annoying error log messages from filling up your syslog file. A big thanks to this ServerFault post on the subject for helping me figure it out.