Force Disconnect iscsi and Reconnect iscsi on centos6+

We’ve all been there, having a webserver which is running low on diskspace. I added an iscsi filesystem to provide sufficient space for creating backups of the system. I host the iscsi filesystem on a FreeNAS box, which usually runs great. But periodically it appears that a bug in the system causes the FreeNAS box to reboot. Then my webservers iscsi connection gets confused and I get to take drastic measures. I force disconnect iscsi and reconnect it.

Force Disconnect iscsi

After ensuring that the FreeNAS box is running properly, I unmount the drive on my webserver.

umount /dev/sda1

I get the details of the existing zombie session with this command.

iscsiadm -m session -o show
 
tcp: [3] 192.168.0.253:3261,2 iqn.2005-10.org.freenas.ctl:tmpfs (non-flash) 

The response shows the details of the active session, the IP address and port, as well as the iqn string. I use them in the following command to force disconnect iscsi.

iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:tmpfs -p 192.168.0.253:3261 -u 
 
Logging out of session [sid: 3, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] 

After getting confirmation that the session has been logged out, I then restart the iscsi service for good measure.

service iscsid restart

Reconnect iscsi

Now that the iscsi service has been restarted the system is ready to reconnect to the iscsi filesystem. Execute the following command to log back in to your iscsi connections.

iscsiadm -m node --login 

Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] successful. 

Once the login has been successful you should be ready to mount your drive again. But I have found that sometimes when reconnecting the filesystem it gives it a new drive identifier. My system will bounce between /dev/sda and /dev/sdb. I use the following command to verify which drive identifier it has used.

Remount your iscsi filesytem

cat /proc/partitions 
 major minor  #blocks  name
 

   11        0    1048575 sr0
  252        0  524288000 vda
  252        1  505411392 vda1
  252        2   18874368 vda2
  252       16  104857600 vdb
  252       17  104857568 vdb1
    8        0 1310720000 sda
    8        1  734002176 sda1 

The last 2 lines show that it connected as /dev/sda where it had been /dev/sdb before it was disconnected. I then change the settings in /etc/fstab to use the new drive identifier. After that you might be good to go, but I have also found when the drive identifier changes that issues with UUID’s can occur. As in my case when I attempted to mount the filesystem.

mount /var/lib/psa/dumps/ 
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so 

That didn’t look very good, so I ran the command it recommended to check dmesg.

dmesg | tail
 [9638158.235890] sd 5:0:0:0: [sda] 2621440000 512-byte logical blocks: (1.34 TB/1.22 TiB)
 [9638158.235895] sd 5:0:0:0: [sda] 32768-byte physical blocks
 [9638158.237537] sd 5:0:0:0: [sda] Write Protect is off
 [9638158.237541] sd 5:0:0:0: [sda] Mode Sense: 7f 00 10 08
 [9638158.238100] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
 [9638158.238193] sd 5:0:0:0: alua: transition timeout set to 60 seconds
 [9638158.238204] sd 5:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
 [9638158.252324]  sda: sda1
 [9638158.256859] sd 5:0:0:0: [sda] Attached SCSI disk
 [9638245.008254] XFS (sda1): Filesystem has duplicate UUID 1615e711-25b3-4120-aa87-2514feebb1c4 - can't mount 

The last line of the output indicated that the filesytem had a duplicate UUID (because it used to be /dev/sdb). XFS tools v5+ have a command to change the UUID of a filesystem. But I was running v3.1.1, so I opted for a different solution. I added an option to the fstab entry that tells the system not to check the UUID.

/dev/sda1 /var/lib/psa/dumps/ xfs _netdev,nouuid 0 0 

Now the filesystem mounts fine, even with a duplicate UUID. This may not be the perfect solution. But when you need to force disconnect iscsi and reconnect iscsi, these commands do the trick for me.

Replacement for Netstat on Linux

Linux distributions grow, get updated, and more advanced. Most tools that you have grown accustomed to using remain, But a few have been replaced. Many reasons for the replacement of core tools. Most of these tools are removed because they are inefficient and replaced with better alternatives. Old networking tools like netstat and ifconfig work by accessing files in the /proc filesystem to aggregate their response. This behavior has been fine for years, especially on smaller systems. As systems grow and complexity increases the chances for inaccurate responses have increased. So what is the replacement for netstat?

The Linux networking subsystem has continued to evolve with the changing of the Linux Kernel, which has lead to new networking functions and processes. The new tools leverage Netlink Sockets to provide their information, which are much more efficient. For a full writeup on the reasons for the switch over I recommend this article on the real reasons of the Linux replacement for netstat.

Netstat is my goto tool for getting a list of open ports on a system and associated processes. Even with the changes in the works I have loaded the program from the net-tools package, until I couldn’t. Recently I was helping a client with a OpenSuse 15 system that didn’t have netstat, and it could not locate the package when I attempted to install it. I then tried the alternative to netstat, ss.

SS is the replacement for netstat

SS stands for “Socket Statistics” and operates in a manner similar to netstat. It formats things differently, leading to some required adjustments. Here is the basic help output for the ss command.

SS Usage

bdoga@webserver:~$ ss --help
 Usage: ss [ OPTIONS ]
        ss [ OPTIONS ] [ FILTER ]
    -h, --help          this message
    -V, --version       output version information
    -n, --numeric       don't resolve service names
    -r, --resolve       resolve host names
    -a, --all           display all sockets
    -l, --listening     display listening sockets
    -o, --options       show timer information
    -e, --extended      show detailed socket information
    -m, --memory        show socket memory usage
    -p, --processes     show process using socket
    -i, --info          show internal TCP information
    -s, --summary       show socket usage summary
    -b, --bpf           show bpf filter socket information
    -E, --events        continually display sockets as they are destroyed
    -Z, --context       display process SELinux security contexts
    -z, --contexts      display process and socket SELinux security contexts
    -N, --net           switch to the specified network namespace name
 

    -4, --ipv4          display only IP version 4 sockets
    -6, --ipv6          display only IP version 6 sockets
    -0, --packet        display PACKET sockets
    -t, --tcp           display only TCP sockets
    -u, --udp           display only UDP sockets
    -d, --dccp          display only DCCP sockets
    -w, --raw           display only RAW sockets
    -x, --unix          display only Unix domain sockets
    -f, --family=FAMILY display sockets of type FAMILY
 

    -A, --query=QUERY, --socket=QUERY
        QUERY := {all|inet|tcp|udp|raw|unix|unix_dgram|unix_stream|unix_seqpacket|packet|netlink}[,QUERY]
 

    -D, --diag=FILE     Dump raw information about TCP sockets to FILE
    -F, --filter=FILE   read filter information from FILE
        FILTER := [ state STATE-FILTER ] [ EXPRESSION ]
        STATE-FILTER := {all|connected|synchronized|bucket|big|TCP-STATES}
          TCP-STATES := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|closed|close-wait|last-ack|listen|closing}
           connected := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
        synchronized := {established|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
              bucket := {syn-recv|time-wait}
                 big := {established|syn-sent|fin-wait-{1,2}|closed|close-wait|last-ack|listen|closing} 

So while it may not be netstat it appears to be a very robust tool. Ss allows the querying of specific information and then filter the results directly. It is useful to filter directly without relying on a tool like grep or awk. But there is nothing to stop you from piping the output into the tool of your choice.

SS Usage Examples

bdoga@webserver:~$ ss -lt
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                    
 LISTEN     0      128        *:6032                     *:*                    
 LISTEN     0      128        *:http                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:ssh                      *:*                    
 LISTEN     0      128        *:https                    *:*                    
 LISTEN     0      128       :::ssh                     :::*               

Show all the tcp ports “-t” that the server is listening “-l” to.

bdoga@webserver:~$ sudo ss -ltp
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("proxysql_galera",pid=28271,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=950,fd=4)) 

Show all the tcp ports “-t” that the server is listening “-l” on. What process “-p” are using those ports. The “-p” flag requires root/sudo access to properly display which processes are using those ports.

bdoga@webserver:~$ sudo ss -lup
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 UNCONN     0      0          *:59879                    *:*                     users:(("rsyslogd",pid=556,fd=5))
 UNCONN     0      0      192.168.28.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=21))
 UNCONN     0      0      10.10.10.28:ntp                *:*                     users:(("ntpd",pid=9421,fd=20))
 UNCONN     0      0      209.33.221.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=19))
 UNCONN     0      0      127.0.0.1:ntp                  *:*                     users:(("ntpd",pid=9421,fd=18))
 UNCONN     0      0          *:ntp                      *:*                     users:(("ntpd",pid=9421,fd=17))
 UNCONN     0      0          *:snmp                     *:*                     users:(("snmpd",pid=1045,fd=7))
 UNCONN     0      0          *:https                    *:*                     users:(("litespeed",pid=21324,fd=18),("litespeed",pid=21321,fd=18))
 UNCONN     0      0      127.0.0.1:8822                 *:*                     users:(("Site24x7Agent",pid=1141,fd=19))
 UNCONN     0      0      fe80::218:3eff:fe57:7bf2%eth1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=24))
 UNCONN     0      0      fe80::216:3cff:fe4e:8673%eth0:ntp                     :::*                     users:(("ntpd",pid=9421,fd=23))
 UNCONN     0      0        ::1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=22))
 UNCONN     0      0         :::ntp                     :::*                     users:(("ntpd",pid=9421,fd=16)) 

Same as the previous example but listing udp ports “-u”.

bdoga@webserver:~$ sudo ss -at4p
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("mysql",pid=28336,fd=28),("timeout",pid=28335,fd=28),("proxysql_galera",pid=28333,fd=28),("proxysql_galera",pid=28332,fd=28),("proxysql_galera",pid=27839,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=22),("timeout",pid=28335,fd=22),("proxysql_galera",pid=28333,fd=22),("proxysql_galera",pid=28332,fd=22),("proxysql_galera",pid=27839,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=21),("timeout",pid=28335,fd=21),("proxysql_galera",pid=28333,fd=21),("proxysql_galera",pid=28332,fd=21),("proxysql_galera",pid=27839,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=20),("timeout",pid=28335,fd=20),("proxysql_galera",pid=28333,fd=20),("proxysql_galera",pid=28332,fd=20),("proxysql_galera",pid=27839,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=19),("timeout",pid=28335,fd=19),("proxysql_galera",pid=28333,fd=19),("proxysql_galera",pid=28332,fd=19),("proxysql_galera",pid=27839,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 TIME-WAIT  0      0      209.33.221.34:58184                209.33.221.83:mysql                
 TIME-WAIT  0      0      127.0.0.1:34804             127.0.0.1:6032                 
 TIME-WAIT  0      0      127.0.0.1:34470             127.0.0.1:6032                 
 TIME-WAIT  0      0      209.33.221.34:45368         209.33.221.36:mysql                
 ESTAB      0      0      209.33.221.34:37194         209.33.221.67:mysql                 users:(("mysql",pid=28336,fd=109),("timeout",pid=28335,fd=109),("proxysql_galera",pid=28333,fd=109),("proxysql_galera",pid=28332,fd=109),("proxysql_galera",pid=27839,fd=109),("proxysql",pid=3649,fd=109))
 TIME-WAIT  0      0      127.0.0.1:33812                127.0.0.1:6032                  

To show all “-a” tcp “-t” but only IPv4 “-4” interfaces and their associated processes “-p”.

I will take a little while to fully make the switch since I have used netstat for decades. Although it appears the future is bright, and ss should be able to take care of me in the future. Hopefully this overview helps you make the switch as well.

How to copy a file into all subdirectories

I was left with a conundrum, I had needed to add a default index.html file into each subdirectory on my web server to ensure that the appropriate response was given if someone browsed to any directory on the web server. The command I came up with to copy the file into all subdirectories was:

ls -d */ | xargs -n 1 cp -i index.html

Where index.html is the file to be copied. The ls -d */ command gets a list of directories in the current directory that can be piped into xargs to execute the copying command, but it wasn’t as robust as I was hoping. This command doesn’t work with directories that have spaces in the name. And it only copies the file into the immediate subdirectories. So I started playing around with the command and came up with the following modified command that will copy a file into all subdirectories recursively.

ls -R | grep ":" | sed "s/^.://" | sed "s/://" | xargs -n 1 cp -n index.html 

Again this command is copying the index.html file, ls -R gets a list of all files and directories recursively from the current folder. grep “:” locates all the directories since they each end with a colon “:”. sed “s/^.://” removes the reference to the current directory “.” in the returned directory list. sed “s/^.://” then cleans off the trailing colons “:” from each directory entry and then the cleaned directories are piped into the xargs command to copy the file into each one.

So give it a go, it could save you a bit of time and hassle now that you can copy a file into all subdirectories.

Reinstall Grub after using CloneZilla

I had made a partition backup of a machine using Clonezilla and wanted to restore it. The restore was successful but because I had only restored the partitions rather than the full disk Grub was not installed in the MBR. Without Grub in the MBR the system failed to boot.

I mounted the new filesystem to /mnt while still using the live Clonezilla disk that I had used for the backup. Then I chrooted using the following command

chroot /mnt

then while in the chroot I attempted to reinstall grub, and since this was a Cloudlinux/Centos install I performed

grub-install /dev/sda

But grub-install complained that it couldn’t find /dev/sda or that /dev/sda was not a valid block device. So then hunting around on the internet for a little bit I came across this article which showed basically how to ensure that your current live

  • /dev
  • /sys
  • /proc

filesystems are accessible inside of your current chroot.

So I ran the following commands outside of the chroot before entering it again.

mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys

Then I chrooted to /mnt again and ran my grub-install command and all was well. The machine booted perfectly after that.