Dump All MySQL Databases into Individual SQL Files

Anyone who is responsible for managing a MySQL database will eventually run into this problem. You either need to dump all your MySQL databases for a backup, or to prepare for an upgrade. Whatever your circumstances are there are several different methods that can be employed to dump your MySQL DBs. Hopefully I can give you the basic tools to get you on your way.

Dump All DBs into a Single SQL File

This first method is what I would call the quick and dirty method. It is straight forward and just dumps all of the databases on a server into a single SQL file. For many individuals this is sufficient, but may not be a good option for larger databases or backup/restore processes. Since all of the DBs are bundled into a single file. Each of the values in brackets “[]” are placeholders for your own values.

mysqldump -u [username] -p --all-databases > allDB.sql

If all you need is to get a quick backup of everything this may be your ticket. The “–all-databases” flag does the magic here, dumping all of the MySQL databases into a single SQL file. More details on this method can be found here. But if you are wanting to be able to easily restore an individual database you may want to use an approach like this.

Dump All MySQL Databases into Individual Files

for I in $(mysql -u [username] -p[mypassword] -h [Hostname/IP] -e 'show databases' -s --skip-column-names); do mysqldump -u [username] -p[mypassword] -h [Hostname/IP] $I > "/home/user/$I.sql"; done

This command first calls “mysql” and gets a list of databases on the server. The command feeds that list of MySQL databases into “mysqldump” to get an individual SQL file for each database on the server. Finally those SQL files are then saved in the location indicated, “/home/user/dbname.sql” in this example.

A few things to note in this command are that first you will notice that I have included the password in the command. The command will complain that using the password on the command line is not safe. However it is required for the command to work. The “-p” is immediately followed by the password, without a space. This is how the option functions, if you add a space it will not work.

The “$I” is a variable and it will have all the database names in it, as it iterates through your listing of DBs. So as you modify the command to fit your specific setup, just make sure to keep that for consistency.

Additional Mysqldump Options

There are a couple of additional mysqldump options that you may want to add depending on your requirements.

--single-transaction

This option keeps mysqldump from trying to get a complete lock on the database. It tells mysqldump to just grab a single transaction and dump the DB contents at the time of that transaction. This can be especially helpful when you are dumping a DB that is especially large, or on a very busy server. Without this option mysqldump may timeout while waiting to get a lock on the DB.

--default-character-set=utf8mb4 

If you are working with data that may contain emoticons you will want this flag. This ensures that your dumped sql file has the correct information to recreate the emoticons when the file gets reimported. Without this option, your emoticons will show up as strange/random text characters when you restore your backup.

Adding Compression

I don’t typically use compression on my SQL dumps since I like to use rdiff-backup as my backup mechanism. But for those who would like to compress their mysqldump in one step here is the basic gist of it.

mysqldump -u [username] -p --all-databases | gzip > allDB.sql.gz

You just pipe the output from the mysqldump command into gzip, or bzip2 to compress the contents. That can be easily added to the command above to dump all MySQL databases into individual files like so.

for I in $(mysql -u [username] -p[mypassword] -h [Hostname/IP] -e 'show databases' -s --skip-column-names); do mysqldump -u [username] -p[mypassword] -h [Hostname/IP] $I | gzip > "/home/user/$I.sql.gz"; done

Hopefully these examples help you get the backups you need. Have some fun along the way.

How to Use Rdiff-backup – A Simply Powerful Backup Tool

In this post I hope to help you understand how to use Rdiff-backup. I stumbled across Rdiff-backup several years ago. It has helped me streamline and simplify most of my file based backup processes from Linux servers. As the name implies, a diff is performed on the files being backed up, so only differences get backed up. As you can probably guess only storing the diffs of changes can lead to a much smaller backup.

Some advantages of Rdiff-backup is that it utilizes rsync, so it can quickly and easily mirror a directory. Backups can happen in seconds if there have been few changes. Your data all travels over a secure ssh connection, so your files are safe during transport. And being a simple command line tool, you can easily script out a backup scenario. Or just use it straight from your terminal.

How I Use Rdiff-backup

I typically use Rdiff-backup with my web hosting clients, it allows for quick backup and restore of their web files. And because most of the the files don’t change from day to day the backup is lightning fast.

rdiff-backup --exclude '**cache/' --exclude '**debug.log' /var/www user@ipAddress::/home/user/backup
rdiff-backup --remove-older-than 52W user@ipAddress::/home/user/backup

Let’s break down the command(s), the first Rdiff-backup command has two “–exclude” options. Those options will as indicated exclude the referenced files or directories from the backup. The exclude options can either have a full relative directory structure or in this case a “**” will match any path. A single asterisk “*” could also be used, but it matches any part of a path not containing a “/” in it.

The “/var/www” part of the command is the directory to backup. Using standard ssh authentication methods “user@ipAddress” ie:”bob@10.1.1.1″ for the login credentials. And then a double colon, which is different from normal rsync/scp formatting. The “::” proceeds the backup destination directory.

So to explain in basic terms, the first command will backup everything under the “/var/www” directory, except any directory ending in “cache/” or file ending in “debug.log”. The backup will be made in the “/home/user/backup” directory.

Backup Lifecycle

Now that you have a backup created how do you manage how long the backup will be kept? Without a command like the second one above, the Rdiff-backup will be kept indefinitely. But adding the second command helps us manage the how long to keep the backups.

rdiff-backup --remove-older-than 52W user@ipAddress::/home/user/backup

This command is a little simpler than the first, it doesn’t specify a source directory. But rather only specifies how long to keep files. The “–remove-older-than” option can take a number of different options. I like to keep my backups for a year, so “52W” gives me 52 weeks of backups. Any existing diffs older than the specified time are removed. Other options are s, m, h, D, W, M, or Y (indicating seconds, minutes, hours, days, weeks, months, or years respectively). Additionally a “B” can be used to indicate the number of backups, ie: “3B” would keep the last three backups.

After specifying the number or timeframe of backups to keep, the only other thing to specify is the backup location. The trimming of backup content is then performed on the given location.

Performing a Restore

So now you have your content backed up, and you need to restore something. A backup is only as good as the data you can restore out of it right? Fortunately the restore process fairly simple as well.

rdiff-backup -r 5D user@ipAddress::/home/user/backup/example.com/index.php /home/localUser/www/restore/

The “-r” option tells Rdiff-backup to restore files, and uses the same time format as the delete option above. In this case we are restoring a version of the file “example.com/index.php” from 5 days ago. The file is being restored/copied to “/home/localUser/www/restore/”. The same can be done for an entire directory structure.

rdiff-backup -r 3B user@ipAddress::/home/user/backup/example.com/ /home/localUser/www/restore/

This command restores/copies all the contents of the example.com directory to “/home/localUser/www/restore/” from 3 Backups ago. Or if you are hunting for a specific day you can always do something like this.

rdiff-backup -r 03-05-2020 user@ipAddress::/home/user/backup/example.com/ /home/localUser/www/restore/

That will perform the same restore, but specifically as of the 5th of March 2020. The date used can be “03/05/2020” or “2020-03-05”, and all indicate midnight as of that day.

For a full rundown on all the options and other details for Rdiff-backup check out the project documentation.

There you have a basic rundown on how to use Rdiff-backup. I find it a very useful and powerful tool, and hope it will help you keep your backups running.

Find Files Not Owned By Specific User or Group

Recently ran into an issue where I needed to search recursively through a file structure. And find files that were not owned by a specific user or group. The issue came up because a client would run a recursive chown on a directory before running git. As a result of the chown they would periodically see the process hang. The chown was to ensure that all the files were writable and wouldn’t gum up the git process. However the hung chown command would cause extra load on the server and lead to system instability.

These files were shared between a few web servers on an NFS share. After some lengthy research I am still stumped as to exactly what the source of the issue is. But it lead me to craft a command that would allow the client to easily check permissions on their files. But without forcing a change to the ownership, which seemed to possibly be the cause of the hanging process.

Find files not owned by specific user

The command was created using some help from this site and this site.

find . ! -group web -or ! -user web -printf "%p - user:%u group:%g\n"

The find command searches from the “.” current directory and recursively checks all files and folders. The “! -group web” section tells the find command to check for files that are NOT “!” owned by the “web” group. Then “! -user web” specifies that it should check for files that are NOT “!” owned by the “web” user. The “-or” tells the find command to match files that are either not owned by the “web” user or group. Changing that to “-and” would only show files that are not owned by the “web” user or group.

The final section ‘-printf “%p – user:%u group:%g\n”‘ tells find how to output the results. “%p” outputs the filename and relative directory structure. “%u” and “%g” output the user and group of the file that is found. Some sample output would look like this.

./logs/access.log.2016_09_30.03.gz - user:root group:web
./logs/access.log.2016_10_04.09.gz - user:root group:web
./logs/access.log.2016_10_06.32.gz - user:root group:web
./logs/access.log.2016_10_04.38.gz - user:root group:web

This command easily helps you determine if your permissions are set correctly. And identify which files will need to have their ownership changed. It is more lightweight and doesn’t force any changes to the filesystem when they are unneeded.

Allow Let’s Encrypt to Bypass HTTP Auth

The story is a common one, you or your client wants to have a site protected by password. But you also want it to be secured with a Let’s Encrypt SSL certificate. The Certificate was easy to provision when you first set up the website. But then you added basic HTTP authentication or IP restrictions to your htaccess file. Now Let’s Encrypt can no longer connect to your site to renew the ssl cert. So how can we allow Let’s Encrypt to bypass HTTP auth and issue your cert?

The following method will work for either IP or basic authentication settings.

How to Allow Let’s Encrypt to Bypass HTTP Auth

Let’s Encrypt needs to access the hidden folder called .well-known contained within your document root. Typically your authentication settings or IP restrictions are in the htaccess file in your document root. This establishes the rules for all child folders. But by default htaccess files found inside child folders can override the parent settings.

By adding a htaccess file in your .well-known directory you can allow Let’s Encrypt to bypass HTTP auth security. Your htaccess file should contain the following:

Order allow,deny
Allow from all
Satisfy Any

These directives tell the web server to allow traffic from any source to access that directory and all child folders.

Thanks to these sites for guiding me to this solution in my own search. You can find their write-ups here and here.

HTTPS Redirect Breaks Let’s Encrypt Renewal

One other issue I have run into that causes problems with the renewal process is your sites HTTPS redirect. In some instances once you add your redirect rule to force all traffic to use https, this can break your Let’s Encrypt Renewal process. And no longer allow the Let’s Encrypt Certbot to validate your site.

Making a slight change to your rewriterule can fix your problem. Your initial rule may have looked something like this:

# Redirection to HTTPS
 RewriteCond %{SERVER_PORT} ^80$ [OR]
 RewriteCond %{HTTPS} =off
 RewriteRule ^(.*)$ https://yoursite.com$1 [R=301,L]

That rule checks for traffic coming in on port 80 or that does not use https and then redirects all of it to the https version of your site.

# Redirection to HTTPS
 RewriteCond %{SERVER_PORT} ^80$ [OR]
 RewriteCond %{HTTPS} =off
 RewriteRule ^(?!/.well-known(?:$|/)).* https://yoursite.com$0 [R=301,L]

By adding this change to the Rewriterule line we have told the server to redirect all traffic that does not start with “/.well-known”. So anything under that directory (Let’s Encrypt) are not redirected to HTTPS.

Thanks to this site for posting their fix so I could find it.

Redirect Only the Root URL Path for Nginx and Apache

A client of mine has a website that was once their primary website. They had about 70+ gigabytes of files for webinars, lectures and ebooks for download on the site. But over time they migrated all their traffic to a new domain with a new design and store. The old domain remained in use but it was used for downloads only. So the question was brought up, how do you redirect only the root URL path of the site. This would allow you to send all the traffic to the root of the domain to their primary site but still allow the downloads.

Redirect only the root URL path for Nginx

Nginx was the natural choice this site with all of the static download content. So I began to search for different options to allow me to redirect only the root url path. While searching I found this page that outlines the location directive to be added into my Nginx config. The directive is this:

location = / {       
     return 301 https://new-site.com/feature-page/; 
}

The “location = /” section tells Nginx to match all queries that only come to the root of the site. And “return 301 https://new-site.com/feature-page/;” tells Nginx to perform a 301 redirect to new-site.com. You can perform a 302 redirect if you prefer by switching the number from 301 to 302.

The directive above can be added in anywhere in your “Server {}” section of the Nginx config.

Redirect only the root URL path for Apache

We don’t want anyone to feel left out. So for all you apache web server lovers out there here is the config to redirect only the root of Apache. I found a good example of how to accomplish this here.

RewriteEngine on 
RewriteCond %{HTTP_HOST} mysite\.com [NC] 
RewriteCond %{REQUEST_URI} ^/$ 
Rewriterule ^(.*)$ http://mysecondsite.com/ [L,R=301]

What do these configuration settings mean? “RewriteEngine on” ensures that Apache’s Rewrite Engine is working. The next line adds a condition to the rewrite rule. It specifies that the request should compare what the server sees as the HTTP_HOST of the request against “mysite.com”. The “[NC]” indicates that comparison should be made in a case insensitive manner.

The next rule checks the REQUEST_URI to see if it matches only the root with nothing after it. When both those conditions are met the request will be redirected to http://mysecondsite.com/. The “[L,R=301]” indicates that the redirect should be a 301 and this is the “Last” rule to check.

So any request to exactly “mysite.com/” will automatically be redirected to “http://mysecondsite.com/”.

How to Fix: “warning: device is not properly aligned” with Parted

Because you are on this page I assume you have received the same warning I did. “warning: device is not properly aligned”. I allocated a new 1 TB iSCSI drive and I attached it to the initiating server. After the iSCSI device was connected I partitioned it to use 100% of the disk. Then I attempted to format the partition with the XFS file system using the following command.

mkfs.xfs /dev/sdb1 

I immediately got the following response:

warning: device is not properly aligned /dev/sdb1
Use -f to force usage of a misaligned device 

Finding the fix for “device is not properly aligned”

So I did some digging and came across this article about how to properly align the partition. I followed the post to grab the following values from the device.

[root@server ~]# cat /sys/block/sdb/queue/optimal_io_size
1048576
[root@server ~]# cat /sys/block/sdb/queue/minimum_io_size
131072
[root@server ~]# cat /sys/block/sdb/alignment_offset
0
[root@server ~]# cat /sys/block/sdb/queue/physical_block_size
131072 

Following the formula that was outlined (optimal_io_size + alignment_offset) / physical_block_size. And got the following results: (1048576 + 0) / 131072 = 8. I plugged that information into parted, which resulted in mixed results.

(parted) mklabel gpt                                                      
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes 
                                                              
(parted) mkpart primary 8s 100%
Warning: You requested a partition from 4096B to 1074GB.                  
The closest location we can manage is 17.4kB to 1074GB.
Is this still acceptable to you?
Yes/No? Yes     
                                                          
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? Ignore 
                                                    
(parted) align-check optimal 1                                            
1 not aligned

Making Sense of the Parted Alignment Response

So my drive was still not properly aligned, but I had some better info which would help me fix the problem. So let’s break down the Parted commands

(parted) mklabel gpt 

Destroyed the existing partition table and cleared things out to start fresh.

(parted) mkpart primary 8s 100%

This command told Parted to create a new primary partition starting at sector 8 and use the rest of the drive. The response that came from this command gave me the clue that I needed to fix the alignment.

Warning: You requested a partition from 4096B to 1074GB.
The closest location we can manage is 17.4kB to 1074GB.

The warning output indicated that 8 sectors of 512B would only come to 4096B or 4kB. But Parted could only adjust the alignment to 17.4kB, so this was still out of alignment as the rest of the output indicated. That gave me the clue I needed to fix it though.

Alignment Fixed

Knowing now that Parted could only get within 17.4kB of the start of my drive I adjusted the sectors to match that. Since 8 sectors was 4096B, I adjusted it to be an even number of sectors beyond the 17.4kB point. Choosing 32768kB, or 64 sectors. This change yielded the following results.

(parted) mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes                                                               

(parted) mkpart primary 64s 100%

(parted) align-check optimal 1
1 aligned

(parted) quit                                                             
Information: You may need to update /etc/fstab.

The partitioning was successful, using the 64 sector starting point pushed it beyond the 17.4kB point. Then it was properly aligned. I formatted the partition with XFS and all went swimmingly.

[root@server ~]# mkfs.xfs /dev/sdb1
 specified blocksize 4096 is less than device physical sector size 131072
 switching to logical sector size 512
 meta-data=/dev/sdb1              isize=256    agcount=32, agsize=8191872 blks
          =                       sectsz=512   attr=2, projid32bit=0
 data     =                       bsize=4096   blocks=262139648, imaxpct=25
          =                       sunit=32     swidth=256 blks
 naming   =version 2              bsize=4096   ascii-ci=0
 log      =internal log           bsize=4096   blocks=128000, version=2
          =                       sectsz=512   sunit=32 blks, lazy-count=1
 realtime =none                   extsz=4096   blocks=0, rtextents=0

Now the device is formatted, mounted, and working like a champ. Who would have known that a simple offset of 32kB could make such a problem.

If you are working with iSCSI then you may find you need to force disconnect and reconnect iSCSI on from time to time. Check out my post on how I handle that.

Rename a file that starts with a hyphen/dash

While working on your *nix based system you may run into this issue from time to time. The file may have a hyphen (-) at the start of the filename because of user error. Or it may have been added by a malicious bit of software. However the hyphen got there it can be a pain to deal with. In this post we will answer how to rename a file that starts with a hyphen.

The first time I encountered a filename like this was while searching the files of a compromised website. The offending malware had created several different new files on the site. And one had been created with a starting hyphen in the filename. This causes issues when you run ‘rm’ to remove the file.

rm -filename.php

The hyphen typically tells a command line utility to anticipate an execution option. So in this case the ‘rm’ command attempts to interpret -filename.php as an option. This would just cause ‘rm’ to return the following on a linux based machine.

rm: invalid option -- 'l'
Try `rm --help' for more information.

Both -f and -i are options for the command so when it gets to ‘l’ it assumes it is just an invalid option. So this becomes an issue when you try to remove or otherwise edit/manipulate a file with a hyphen in the name.

Dealing with the hyphenated filename

Fortunately there is a relatively easy method to deal with a file that starts with a hyphen/dash. Adding a double dash (–) before the filename will fix it.

 rm -- -filename.php

This behavior should be universal for GNU/Linux commands. The double dash/double hyphen tells the command that no more command line options will be given. As a result your favorite command will ignore any further dashes and allow you to mv/cp/rm the file.

So now you know how to rename a file that starts with a hyphen on the command line. Hopefully that makes your life a little better.

I originally learned of this technique from this helpful post on superuser.com.

Want to learn how to recursively delete specific files, check out this post to find out how.

How to Recursively Delete Specific Files

Why did I need to recursively delete specific files you ask? Well, recently one of my clients websites was attacked by malware. As part of the attack the malware had added several files throughout the website file structure with a common filename or extension. I found that all the added files had a common .php5 extension. Looking through the file structure revealed a large number of files that needed removal. Some digging revealed a few linux command line shortcuts to speed up your malware recovery or just remove unwanted files. Putting find to work to remove the unwanted files with a single command.

Warning: These Commands Remove Files Permanently

Before using any of the following commands realize that they are going to delete your files. And as cool as I think they are I have no idea what your setup looks like. So before you run any of them, make sure you have a good, complete, current backup of your data.

Delete files with specific extensions

To remove all the files with a specific file extension, the following examples will set you up. This command will search from the current directory and delete all files with a php5 extension in all subdirectories.

find . -type f -name '*.php5' -delete

Or for clarities sake

find . -type f -name '*.[add your extension here]' -delete

Delete files with a specific filename

If it is a specific filename you are wanting to delete recursively just remove the ‘*.extension’ and replace it with the filename.

find . -type f -name '[insert filename here]' -delete

The following extension will delete all files named “debug.log” recursively through the current folder.

find . -type f -name 'debug.log' -delete

Or maybe you are wanting to delete all the access log files for a nginx from 2018.

find /var/log/nginx/ -type f -name 'access.log-2018*.log' -delete

Usually these commands do what I want. But sometimes I need a command that also has a timeframe

Delete files added within a specific timeframe

After removing all the files with a specific name or extension I decided I needed one more step. So I hunted for a way to find all the files added within the last week. The client discovered the attack just over a week since the last update.

The following command will delete any files beneath the current directory that have been modified within the last 24 hours.

find . -mtime -1 -delete

This command will delete any files beneath the current directory that have been modified within the last 48 hours.

find . -mtime +1 -delete

And this command will delete any files beneath the current directory that have been modified within the last 96 hours. Each additional number adds 24 hours to the check 4, 5, etc. With the first 24 hours implied.

find . -mtime +3 -delete

Combine the two for specific files within a specific timeframe

By combining the commands above you can delete any files ending in .log. Which also sit beneath the current directory that have been modified within the last 96 hours.

find . -mtime +3 -type f -name '*.log' -delete

Those commands saved me a ton of time, and got my client back up and running. Hopefully they can do the same for you.

Here are some of the resources that I used when researching this topic:

Want to learn how to copy a file into all subdirectories, check out this post to find out how.

Force Disconnect iscsi and Reconnect iscsi on centos6+

We’ve all been there, having a webserver which is running low on diskspace. I added an iscsi filesystem to provide sufficient space for creating backups of the system. I host the iscsi filesystem on a FreeNAS box, which usually runs great. But periodically it appears that a bug in the system causes the FreeNAS box to reboot. Then my webservers iscsi connection gets confused and I get to take drastic measures. I force disconnect iscsi and reconnect it.

Force Disconnect iscsi

After ensuring that the FreeNAS box is running properly, I unmount the drive on my webserver.

umount /dev/sda1

I get the details of the existing zombie session with this command.

iscsiadm -m session -o show
 
tcp: [3] 192.168.0.253:3261,2 iqn.2005-10.org.freenas.ctl:tmpfs (non-flash) 

The response shows the details of the active session, the IP address and port, as well as the iqn string. I use them in the following command to force disconnect iscsi.

iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:tmpfs -p 192.168.0.253:3261 -u 
 
Logging out of session [sid: 3, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] 

After getting confirmation that the session has been logged out, I then restart the iscsi service for good measure.

service iscsid restart

Reconnect iscsi

Now that the iscsi service has been restarted the system is ready to reconnect to the iscsi filesystem. Execute the following command to log back in to your iscsi connections.

iscsiadm -m node --login 

Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] successful. 

Once the login has been successful you should be ready to mount your drive again. But I have found that sometimes when reconnecting the filesystem it gives it a new drive identifier. My system will bounce between /dev/sda and /dev/sdb. I use the following command to verify which drive identifier it has used.

Remount your iscsi filesytem

cat /proc/partitions 
 major minor  #blocks  name
 

   11        0    1048575 sr0
  252        0  524288000 vda
  252        1  505411392 vda1
  252        2   18874368 vda2
  252       16  104857600 vdb
  252       17  104857568 vdb1
    8        0 1310720000 sda
    8        1  734002176 sda1 

The last 2 lines show that it connected as /dev/sda where it had been /dev/sdb before it was disconnected. I then change the settings in /etc/fstab to use the new drive identifier. After that you might be good to go, but I have also found when the drive identifier changes that issues with UUID’s can occur. As in my case when I attempted to mount the filesystem.

mount /var/lib/psa/dumps/ 
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so 

That didn’t look very good, so I ran the command it recommended to check dmesg.

dmesg | tail
 [9638158.235890] sd 5:0:0:0: [sda] 2621440000 512-byte logical blocks: (1.34 TB/1.22 TiB)
 [9638158.235895] sd 5:0:0:0: [sda] 32768-byte physical blocks
 [9638158.237537] sd 5:0:0:0: [sda] Write Protect is off
 [9638158.237541] sd 5:0:0:0: [sda] Mode Sense: 7f 00 10 08
 [9638158.238100] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
 [9638158.238193] sd 5:0:0:0: alua: transition timeout set to 60 seconds
 [9638158.238204] sd 5:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
 [9638158.252324]  sda: sda1
 [9638158.256859] sd 5:0:0:0: [sda] Attached SCSI disk
 [9638245.008254] XFS (sda1): Filesystem has duplicate UUID 1615e711-25b3-4120-aa87-2514feebb1c4 - can't mount 

The last line of the output indicated that the filesytem had a duplicate UUID (because it used to be /dev/sdb). XFS tools v5+ have a command to change the UUID of a filesystem. But I was running v3.1.1, so I opted for a different solution. I added an option to the fstab entry that tells the system not to check the UUID.

/dev/sda1 /var/lib/psa/dumps/ xfs _netdev,nouuid 0 0 

Now the filesystem mounts fine, even with a duplicate UUID. This may not be the perfect solution. But when you need to force disconnect iscsi and reconnect iscsi, these commands do the trick for me.

Here are some of the resources that I used when researching this topic:

Want to learn how to find a Replacement for Netstat on Linux, check out this post to find out how.

Replacement for Netstat on Linux

Linux distributions grow, get updated, and more advanced. Most tools that you have grown accustomed to using remain, But a few have been replaced. Many reasons for the replacement of core tools. Most of these tools are removed because they are inefficient and replaced with better alternatives. Old networking tools like netstat and ifconfig work by accessing files in the /proc filesystem to aggregate their response. This behavior has been fine for years, especially on smaller systems. As systems grow and complexity increases the chances for inaccurate responses have increased. So what is the replacement for netstat?

The Linux networking subsystem has continued to evolve with the changing of the Linux Kernel, which has lead to new networking functions and processes. The new tools leverage Netlink Sockets to provide their information, which are much more efficient. For a full writeup on the reasons for the switch over I recommend this article on the real reasons of the Linux replacement for netstat.

Netstat is my goto tool for getting a list of open ports on a system and associated processes. Even with the changes in the works I have loaded the program from the net-tools package, until I couldn’t. Recently I was helping a client with a OpenSuse 15 system that didn’t have netstat, and it could not locate the package when I attempted to install it. I then tried the alternative to netstat, ss.

SS is the replacement for netstat

SS stands for “Socket Statistics” and operates in a manner similar to netstat. It formats things differently, leading to some required adjustments. Here is the basic help output for the ss command.

SS Usage

bdoga@webserver:~$ ss --help
 Usage: ss [ OPTIONS ]
        ss [ OPTIONS ] [ FILTER ]
    -h, --help          this message
    -V, --version       output version information
    -n, --numeric       don't resolve service names
    -r, --resolve       resolve host names
    -a, --all           display all sockets
    -l, --listening     display listening sockets
    -o, --options       show timer information
    -e, --extended      show detailed socket information
    -m, --memory        show socket memory usage
    -p, --processes     show process using socket
    -i, --info          show internal TCP information
    -s, --summary       show socket usage summary
    -b, --bpf           show bpf filter socket information
    -E, --events        continually display sockets as they are destroyed
    -Z, --context       display process SELinux security contexts
    -z, --contexts      display process and socket SELinux security contexts
    -N, --net           switch to the specified network namespace name
 

    -4, --ipv4          display only IP version 4 sockets
    -6, --ipv6          display only IP version 6 sockets
    -0, --packet        display PACKET sockets
    -t, --tcp           display only TCP sockets
    -u, --udp           display only UDP sockets
    -d, --dccp          display only DCCP sockets
    -w, --raw           display only RAW sockets
    -x, --unix          display only Unix domain sockets
    -f, --family=FAMILY display sockets of type FAMILY
 

    -A, --query=QUERY, --socket=QUERY
        QUERY := {all|inet|tcp|udp|raw|unix|unix_dgram|unix_stream|unix_seqpacket|packet|netlink}[,QUERY]
 

    -D, --diag=FILE     Dump raw information about TCP sockets to FILE
    -F, --filter=FILE   read filter information from FILE
        FILTER := [ state STATE-FILTER ] [ EXPRESSION ]
        STATE-FILTER := {all|connected|synchronized|bucket|big|TCP-STATES}
          TCP-STATES := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|closed|close-wait|last-ack|listen|closing}
           connected := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
        synchronized := {established|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
              bucket := {syn-recv|time-wait}
                 big := {established|syn-sent|fin-wait-{1,2}|closed|close-wait|last-ack|listen|closing} 

So while it may not be netstat it appears to be a very robust tool. Ss allows the querying of specific information and then filter the results directly. It is useful to filter directly without relying on a tool like grep or awk. But there is nothing to stop you from piping the output into the tool of your choice.

SS Usage Examples

bdoga@webserver:~$ ss -lt
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                    
 LISTEN     0      128        *:6032                     *:*                    
 LISTEN     0      128        *:http                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:ssh                      *:*                    
 LISTEN     0      128        *:https                    *:*                    
 LISTEN     0      128       :::ssh                     :::*               

Show all the tcp ports “-t” that the server is listening “-l” to.

bdoga@webserver:~$ sudo ss -ltp
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("proxysql_galera",pid=28271,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=950,fd=4)) 

Show all the tcp ports “-t” that the server is listening “-l” on. What process “-p” are using those ports. The “-p” flag requires root/sudo access to properly display which processes are using those ports.

bdoga@webserver:~$ sudo ss -lup
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 UNCONN     0      0          *:59879                    *:*                     users:(("rsyslogd",pid=556,fd=5))
 UNCONN     0      0      192.168.28.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=21))
 UNCONN     0      0      10.10.10.28:ntp                *:*                     users:(("ntpd",pid=9421,fd=20))
 UNCONN     0      0      209.33.221.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=19))
 UNCONN     0      0      127.0.0.1:ntp                  *:*                     users:(("ntpd",pid=9421,fd=18))
 UNCONN     0      0          *:ntp                      *:*                     users:(("ntpd",pid=9421,fd=17))
 UNCONN     0      0          *:snmp                     *:*                     users:(("snmpd",pid=1045,fd=7))
 UNCONN     0      0          *:https                    *:*                     users:(("litespeed",pid=21324,fd=18),("litespeed",pid=21321,fd=18))
 UNCONN     0      0      127.0.0.1:8822                 *:*                     users:(("Site24x7Agent",pid=1141,fd=19))
 UNCONN     0      0      fe80::218:3eff:fe57:7bf2%eth1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=24))
 UNCONN     0      0      fe80::216:3cff:fe4e:8673%eth0:ntp                     :::*                     users:(("ntpd",pid=9421,fd=23))
 UNCONN     0      0        ::1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=22))
 UNCONN     0      0         :::ntp                     :::*                     users:(("ntpd",pid=9421,fd=16)) 

Same as the previous example but listing udp ports “-u”.

bdoga@webserver:~$ sudo ss -at4p
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("mysql",pid=28336,fd=28),("timeout",pid=28335,fd=28),("proxysql_galera",pid=28333,fd=28),("proxysql_galera",pid=28332,fd=28),("proxysql_galera",pid=27839,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=22),("timeout",pid=28335,fd=22),("proxysql_galera",pid=28333,fd=22),("proxysql_galera",pid=28332,fd=22),("proxysql_galera",pid=27839,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=21),("timeout",pid=28335,fd=21),("proxysql_galera",pid=28333,fd=21),("proxysql_galera",pid=28332,fd=21),("proxysql_galera",pid=27839,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=20),("timeout",pid=28335,fd=20),("proxysql_galera",pid=28333,fd=20),("proxysql_galera",pid=28332,fd=20),("proxysql_galera",pid=27839,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=19),("timeout",pid=28335,fd=19),("proxysql_galera",pid=28333,fd=19),("proxysql_galera",pid=28332,fd=19),("proxysql_galera",pid=27839,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 TIME-WAIT  0      0      209.33.221.34:58184                209.33.221.83:mysql                
 TIME-WAIT  0      0      127.0.0.1:34804             127.0.0.1:6032                 
 TIME-WAIT  0      0      127.0.0.1:34470             127.0.0.1:6032                 
 TIME-WAIT  0      0      209.33.221.34:45368         209.33.221.36:mysql                
 ESTAB      0      0      209.33.221.34:37194         209.33.221.67:mysql                 users:(("mysql",pid=28336,fd=109),("timeout",pid=28335,fd=109),("proxysql_galera",pid=28333,fd=109),("proxysql_galera",pid=28332,fd=109),("proxysql_galera",pid=27839,fd=109),("proxysql",pid=3649,fd=109))
 TIME-WAIT  0      0      127.0.0.1:33812                127.0.0.1:6032                  

To show all “-a” tcp “-t” but only IPv4 “-4” interfaces and their associated processes “-p”.

I will take a little while to fully make the switch since I have used netstat for decades. Although it appears the future is bright, and ss should be able to take care of me in the future. Hopefully this overview helps you make the switch as well.

Here are some of the resources that I used when researching this topic:

Find Open Ports in Linux

Want to learn how to Force Disconnect iscsi and Reconnect iscsi on centos6+, check out this post to find out how.