Force Disconnect iscsi and Reconnect iscsi on centos6+

We’ve all been there, having a webserver which is running low on diskspace. I added an iscsi filesystem to provide sufficient space for creating backups of the system. I host the iscsi filesystem on a FreeNAS box, which usually runs great. But periodically it appears that a bug in the system causes the FreeNAS box to reboot. Then my webservers iscsi connection gets confused and I get to take drastic measures. I force disconnect iscsi and reconnect it.

Force Disconnect iscsi

After ensuring that the FreeNAS box is running properly, I unmount the drive on my webserver.

umount /dev/sda1

I get the details of the existing zombie session with this command.

iscsiadm -m session -o show
tcp: [3],2 (non-flash) 

The response shows the details of the active session, the IP address and port, as well as the iqn string. I use them in the following command to force disconnect iscsi.

iscsiadm -m node -T -p -u 
Logging out of session [sid: 3, target:, portal:,3261] 

After getting confirmation that the session has been logged out, I then restart the iscsi service for good measure.

service iscsid restart

Reconnect iscsi

Now that the iscsi service has been restarted the system is ready to reconnect to the iscsi filesystem. Execute the following command to log back in to your iscsi connections.

iscsiadm -m node --login 

Logging in to [iface: default, target:, portal:,3261] (multiple)
Login to [iface: default, target:, portal:,3261] successful. 

Once the login has been successful you should be ready to mount your drive again. But I have found that sometimes when reconnecting the filesystem it gives it a new drive identifier. My system will bounce between /dev/sda and /dev/sdb. I use the following command to verify which drive identifier it has used.

Remount your iscsi filesytem

cat /proc/partitions 
 major minor  #blocks  name

   11        0    1048575 sr0
  252        0  524288000 vda
  252        1  505411392 vda1
  252        2   18874368 vda2
  252       16  104857600 vdb
  252       17  104857568 vdb1
    8        0 1310720000 sda
    8        1  734002176 sda1 

The last 2 lines show that it connected as /dev/sda where it had been /dev/sdb before it was disconnected. I then change the settings in /etc/fstab to use the new drive identifier. After that you might be good to go, but I have also found when the drive identifier changes that issues with UUID’s can occur. As in my case when I attempted to mount the filesystem.

mount /var/lib/dumps/ 
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so 

That didn’t look very good, so I ran the command it recommended to check dmesg.

dmesg | tail
 [9638158.235890] sd 5:0:0:0: [sda] 2621440000 512-byte logical blocks: (1.34 TB/1.22 TiB)
 [9638158.235895] sd 5:0:0:0: [sda] 32768-byte physical blocks
 [9638158.237537] sd 5:0:0:0: [sda] Write Protect is off
 [9638158.237541] sd 5:0:0:0: [sda] Mode Sense: 7f 00 10 08
 [9638158.238100] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
 [9638158.238193] sd 5:0:0:0: alua: transition timeout set to 60 seconds
 [9638158.238204] sd 5:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
 [9638158.252324]  sda: sda1
 [9638158.256859] sd 5:0:0:0: [sda] Attached SCSI disk
 [9638245.008254] XFS (sda1): Filesystem has duplicate UUID 1615e711-25b3-4120-aa87-2514feebb1c4 - can't mount 

The last line of the output indicated that the filesytem had a duplicate UUID (because it used to be /dev/sdb). XFS tools v5+ have a command to change the UUID of a filesystem. But I was running v3.1.1, so I opted for a different solution. I added an option to the fstab entry that tells the system not to check the UUID.

/dev/sda1 /var/lib/dumps/ xfs _netdev,nouuid 0 0 

Now the filesystem mounts fine, even with a duplicate UUID. This may not be the perfect solution. But when you need to force disconnect iscsi and reconnect iscsi, these commands do the trick for me.

Here are some of the resources that I used when researching this topic:

Want to learn how to find a Replacement for Netstat on Linux, check out this post to find out how.

Replacement for Netstat on Linux

Linux distributions grow, get updated, and more advanced. Most tools that you have grown accustomed to using remain, But a few have been replaced. Many reasons for the replacement of core tools. Most of these tools are removed because they are inefficient and replaced with better alternatives. Old networking tools like netstat and ifconfig work by accessing files in the /proc filesystem to aggregate their response. This behavior has been fine for years, especially on smaller systems. As systems grow and complexity increases the chances for inaccurate responses have increased. So what is the replacement for netstat?

The Linux networking subsystem has continued to evolve with the changing of the Linux Kernel, which has lead to new networking functions and processes. The new tools leverage Netlink Sockets to provide their information, which are much more efficient. For a full writeup on the reasons for the switch over I recommend this article on the real reasons of the Linux replacement for netstat.

Netstat is my goto tool for getting a list of open ports on a system and associated processes. Even with the changes in the works I have loaded the program from the net-tools package, until I couldn’t. Recently I was helping a client with a OpenSuse 15 system that didn’t have netstat, and it could not locate the package when I attempted to install it. I then tried the alternative to netstat, ss.

SS is the replacement for netstat

SS stands for “Socket Statistics” and operates in a manner similar to netstat. It formats things differently, leading to some required adjustments. Here is the basic help output for the ss command.

SS Usage

bdoga@webserver:~$ ss --help
 Usage: ss [ OPTIONS ]
        ss [ OPTIONS ] [ FILTER ]
    -h, --help          this message
    -V, --version       output version information
    -n, --numeric       don't resolve service names
    -r, --resolve       resolve host names
    -a, --all           display all sockets
    -l, --listening     display listening sockets
    -o, --options       show timer information
    -e, --extended      show detailed socket information
    -m, --memory        show socket memory usage
    -p, --processes     show process using socket
    -i, --info          show internal TCP information
    -s, --summary       show socket usage summary
    -b, --bpf           show bpf filter socket information
    -E, --events        continually display sockets as they are destroyed
    -Z, --context       display process SELinux security contexts
    -z, --contexts      display process and socket SELinux security contexts
    -N, --net           switch to the specified network namespace name

    -4, --ipv4          display only IP version 4 sockets
    -6, --ipv6          display only IP version 6 sockets
    -0, --packet        display PACKET sockets
    -t, --tcp           display only TCP sockets
    -u, --udp           display only UDP sockets
    -d, --dccp          display only DCCP sockets
    -w, --raw           display only RAW sockets
    -x, --unix          display only Unix domain sockets
    -f, --family=FAMILY display sockets of type FAMILY

    -A, --query=QUERY, --socket=QUERY
        QUERY := {all|inet|tcp|udp|raw|unix|unix_dgram|unix_stream|unix_seqpacket|packet|netlink}[,QUERY]

    -D, --diag=FILE     Dump raw information about TCP sockets to FILE
    -F, --filter=FILE   read filter information from FILE
        FILTER := [ state STATE-FILTER ] [ EXPRESSION ]
        STATE-FILTER := {all|connected|synchronized|bucket|big|TCP-STATES}
          TCP-STATES := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|closed|close-wait|last-ack|listen|closing}
           connected := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
        synchronized := {established|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
              bucket := {syn-recv|time-wait}
                 big := {established|syn-sent|fin-wait-{1,2}|closed|close-wait|last-ack|listen|closing} 

So while it may not be netstat it appears to be a very robust tool. Ss allows the querying of specific information and then filter the results directly. It is useful to filter directly without relying on a tool like grep or awk. But there is nothing to stop you from piping the output into the tool of your choice.

SS Usage Examples

bdoga@webserver:~$ ss -lt
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                    
 LISTEN     0      128        *:6032                     *:*                    
 LISTEN     0      128        *:http                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:ssh                      *:*                    
 LISTEN     0      128        *:https                    *:*                    
 LISTEN     0      128       :::ssh                     :::*               

Show all the tcp ports “-t” that the server is listening “-l” to.

bdoga@webserver:~$ sudo ss -ltp
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("proxysql_galera",pid=28271,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=950,fd=4)) 

Show all the tcp ports “-t” that the server is listening “-l” on. What process “-p” are using those ports. The “-p” flag requires root/sudo access to properly display which processes are using those ports.

bdoga@webserver:~$ sudo ss -lup
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 UNCONN     0      0          *:59879                    *:*                     users:(("rsyslogd",pid=556,fd=5))
 UNCONN     0      0              *:*                     users:(("ntpd",pid=9421,fd=21))
 UNCONN     0      0                *:*                     users:(("ntpd",pid=9421,fd=20))
 UNCONN     0      0              *:*                     users:(("ntpd",pid=9421,fd=19))
 UNCONN     0      0                  *:*                     users:(("ntpd",pid=9421,fd=18))
 UNCONN     0      0          *:ntp                      *:*                     users:(("ntpd",pid=9421,fd=17))
 UNCONN     0      0          *:snmp                     *:*                     users:(("snmpd",pid=1045,fd=7))
 UNCONN     0      0          *:https                    *:*                     users:(("litespeed",pid=21324,fd=18),("litespeed",pid=21321,fd=18))
 UNCONN     0      0                 *:*                     users:(("Site24x7Agent",pid=1141,fd=19))
 UNCONN     0      0      fe80::218:3eff:fe57:7bf2%eth1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=24))
 UNCONN     0      0      fe80::216:3cff:fe4e:8673%eth0:ntp                     :::*                     users:(("ntpd",pid=9421,fd=23))
 UNCONN     0      0        ::1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=22))
 UNCONN     0      0         :::ntp                     :::*                     users:(("ntpd",pid=9421,fd=16)) 

Same as the previous example but listing udp ports “-u”.

bdoga@webserver:~$ sudo ss -at4p
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("mysql",pid=28336,fd=28),("timeout",pid=28335,fd=28),("proxysql_galera",pid=28333,fd=28),("proxysql_galera",pid=28332,fd=28),("proxysql_galera",pid=27839,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=22),("timeout",pid=28335,fd=22),("proxysql_galera",pid=28333,fd=22),("proxysql_galera",pid=28332,fd=22),("proxysql_galera",pid=27839,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=21),("timeout",pid=28335,fd=21),("proxysql_galera",pid=28333,fd=21),("proxysql_galera",pid=28332,fd=21),("proxysql_galera",pid=27839,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=20),("timeout",pid=28335,fd=20),("proxysql_galera",pid=28333,fd=20),("proxysql_galera",pid=28332,fd=20),("proxysql_galera",pid=27839,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=19),("timeout",pid=28335,fd=19),("proxysql_galera",pid=28333,fd=19),("proxysql_galera",pid=28332,fd=19),("proxysql_galera",pid=27839,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 TIME-WAIT  0      0                      
 TIME-WAIT  0      0                    
 TIME-WAIT  0      0                    
 TIME-WAIT  0      0                
 ESTAB      0      0                 users:(("mysql",pid=28336,fd=109),("timeout",pid=28335,fd=109),("proxysql_galera",pid=28333,fd=109),("proxysql_galera",pid=28332,fd=109),("proxysql_galera",pid=27839,fd=109),("proxysql",pid=3649,fd=109))
 TIME-WAIT  0      0                        

To show all “-a” tcp “-t” but only IPv4 “-4” interfaces and their associated processes “-p”.

I will take a little while to fully make the switch since I have used netstat for decades. Although it appears the future is bright, and ss should be able to take care of me in the future. Hopefully this overview helps you make the switch as well.

Here are some of the resources that I used when researching this topic:

Find Open Ports in Linux

Want to learn how to Force Disconnect iscsi and Reconnect iscsi on centos6+, check out this post to find out how.

Recursive Find and Replace on the Command Line

The Problem

Recently I moved a website that used a ton of legacy php code from the clients production server to a development location. After the move was complete I found that the previous developers had been extremely sloppy. Rather than having a single location/file for DB credentials, they had it in 4 places.

After I figured out where all the locations for DB credentials were I started getting Open_Basedir errors. The original developer had hard coded the web root location hundreds of times in hundreds of files. For just a moment I felt just a bit overwhelmed, then I remembered that I have the terminal to solve problems like this.

The Solution: Recursive Find and Replace on the Command Line

After a bit of research I came up with the following command to recursively search through the entire codebase. When an instance the old web root is detected it replaces it with the correct one.

grep -rl [search for string] . | xargs sed -i s@[search for string]@[replace with string]@g

Or another example with actual search/replace strings

grep -rl /var/www/vhosts/ . | xargs sed -i s@/var/www/vhosts/

That command breaks down in the following manner.

“grep -rl” searches recursively for the string you specified “/var/www/vhosts/” starting in the current directory “.”, the “-r” option specifies the recursive search, and “-l” specifies that the system should return only the filenames that contain the string.

those results are then piped “| ” into “sed”, the “-i” option specifies that it should make the changes in place. Then the find replace sequence in this case “s@[search for string]@[replace with string]@g”. The “@” signs could be almost any other value, typically they are a “/” but in this case the strings to find and replace both had “/” in each one so it wouldn’t work as the bordering character. So replacing the “/” with “@” helps SED keep on track. It could easily have been a “#” or “$”, just use what you need to depending on your string.

And with that I was home free no more Open_Basedir issues. Thanks to the command line recursive find and replace all those entries didn’t have to be done by hand.

Thanks Linux Shell

Here are some of the resources that I used when researching this topic:

Find and Replace string in all files recursive using grep and sed

Now you’ve learned to perform a command line recursive find and replace, why not read this post. Learn how to recursively delete specific files using the command line.

How to copy a file into all subdirectories

A project I was working on required adding a default index.html file into each subdirectory in a directory. This ensured an appropriate response was given when someone browsed to any directory on the web server. The command I came up with to copy a file into all subdirectories was:

ls -d */ | xargs -n 1 cp -i index.html

The command breaks down like this. Where Index.html is the file that will be copied. The ‘ls -d */’ command retrieves a list of directories in the current directory. The list is piped ‘|’ into xargs to execute the copying process. Unfortunately this command was not as robust as I was hoping. It doesn’t work with directories that have spaces in the name. It also only copies the file into the immediate subdirectories. Noting those issues I refined the command. The resulting modified command will copy a file into all subdirectories recursively.

ls -R | grep ":" | sed "s/^.://" | sed "s/://" | xargs -n 1 cp -n index.html 

This command also copies the index.html file. ‘ls -R’ retrieves a list of all files and directories recursively from the current folder. The ‘grep “:”‘ portion locates all the directories since they each end with a colon “:”. Then the ‘sed “s/^.://”‘ section removes the reference to the current directory “.” in the returned directory list. The ‘sed “s/^.://”‘ portion cleans off the trailing colons “:” from each directory entry. The resulting cleaned list of directories is piped into the xargs command to copy the file into each one.

So give it a go, it could save you a bit of time and hassle now that you can copy a file into all subdirectories.

I created this post using these resources:

How to copy a file to multiple directories using the gnu cp command

Now you’ve learned to recursively copy a file into all subdirectories, why not read this post. Learn how to recursively delete specific files using the command line.

Reinstall Grub after using CloneZilla

I had made a partition backup of a machine using Clonezilla and wanted to restore it. The restore was successful but because I had only restored the partitions rather than the full disk Grub was not installed in the MBR. Without Grub in the MBR the system failed to boot.

I mounted the new filesystem to /mnt while still using the live Clonezilla disk that I had used for the backup. Then I chrooted using the following command

chroot /mnt

then while in the chroot I attempted to reinstall grub, and since this was a Cloudlinux/Centos install I performed

grub-install /dev/sda

But grub-install complained that it couldn’t find /dev/sda or that /dev/sda was not a valid block device. So then hunting around on the internet for a little bit I came across this article which showed basically how to ensure that your current live

  • /dev
  • /sys
  • /proc

filesystems are accessible inside of your current chroot.

So I ran the following commands outside of the chroot before entering it again.

mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys

Then I chrooted to /mnt again and ran my grub-install command and all was well. The machine booted perfectly after that.

Shameless Plug

If you need to repartition or reformat a drive. Or resize partitions and check the health of your drive for Linux or Windows based machines give Parted Magic a try. It takes your favorite open source tools and puts them in an easy to use bootable linux CD. It has become my goto tool for all hard drive management tasks.