We’ve all been there, having a webserver which is running low on diskspace. I added an iscsi filesystem to provide sufficient space for creating backups of the system. I host the iscsi filesystem on a FreeNAS box, which usually runs great. But periodically it appears that a bug in the system causes the FreeNAS box to reboot. Then my webservers iscsi connection gets confused and I get to take drastic measures. I force disconnect iscsi and reconnect it.
Force Disconnect iscsi
After ensuring that the FreeNAS box is running properly, I unmount the drive on my webserver.
I get the details of the existing zombie session with this command.
iscsiadm -m session -o show
tcp:  192.168.0.253:3261,2 iqn.2005-10.org.freenas.ctl:tmpfs (non-flash)
The response shows the details of the active session, the IP address and port, as well as the iqn string. I use them in the following command to force disconnect iscsi.
iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:tmpfs -p 192.168.0.253:3261 -u
Logging out of session [sid: 3, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261]
After getting confirmation that the session has been logged out, I then restart the iscsi service for good measure.
service iscsid restart
Now that the iscsi service has been restarted the system is ready to reconnect to the iscsi filesystem. Execute the following command to log back in to your iscsi connections.
iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] successful.
Once the login has been successful you should be ready to mount your drive again. But I have found that sometimes when reconnecting the filesystem it gives it a new drive identifier. My system will bounce between /dev/sda and /dev/sdb. I use the following command to verify which drive identifier it has used.
Remount your iscsi filesytem
major minor #blocks name
11 0 1048575 sr0
252 0 524288000 vda
252 1 505411392 vda1
252 2 18874368 vda2
252 16 104857600 vdb
252 17 104857568 vdb1
8 0 1310720000 sda
8 1 734002176 sda1
The last 2 lines show that it connected as /dev/sda where it had been /dev/sdb before it was disconnected. I then change the settings in /etc/fstab to use the new drive identifier. After that you might be good to go, but I have also found when the drive identifier changes that issues with UUID’s can occur. As in my case when I attempted to mount the filesystem.
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
That didn’t look very good, so I ran the command it recommended to check dmesg.
dmesg | tail
[9638158.235890] sd 5:0:0:0: [sda] 2621440000 512-byte logical blocks: (1.34 TB/1.22 TiB)
[9638158.235895] sd 5:0:0:0: [sda] 32768-byte physical blocks
[9638158.237537] sd 5:0:0:0: [sda] Write Protect is off
[9638158.237541] sd 5:0:0:0: [sda] Mode Sense: 7f 00 10 08
[9638158.238100] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[9638158.238193] sd 5:0:0:0: alua: transition timeout set to 60 seconds
[9638158.238204] sd 5:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
[9638158.252324] sda: sda1
[9638158.256859] sd 5:0:0:0: [sda] Attached SCSI disk
[9638245.008254] XFS (sda1): Filesystem has duplicate UUID 1615e711-25b3-4120-aa87-2514feebb1c4 - can't mount
The last line of the output indicated that the filesytem had a duplicate UUID (because it used to be /dev/sdb). XFS tools v5+ have a command to change the UUID of a filesystem. But I was running v3.1.1, so I opted for a different solution. I added an option to the fstab entry that tells the system not to check the UUID.
/dev/sda1 /var/lib/dumps/ xfs _netdev,nouuid 0 0
Now the filesystem mounts fine, even with a duplicate UUID. This may not be the perfect solution. But when you need to force disconnect iscsi and reconnect iscsi, these commands do the trick for me.
Here are some of the resources that I used when researching this topic:
Want to learn how to find a Replacement for Netstat on Linux, check out this post to find out how.