How to Recursively Delete Specific Files

Why did I need to recursively delete specific files you ask? Well, recently one of my clients websites was attacked by malware. As part of the attack the malware had added several files throughout the website file structure with a common filename or extension. I found that all the added files had a common .php5 extension. Looking through the file structure revealed a large number of files that needed removal. Some digging revealed a few linux command line shortcuts to speed up your malware recovery or just remove unwanted files. Putting find to work to remove the unwanted files with a single command. For some of the source pages look here and here.

Warning: These Commands Remove Files Permanently

Before using any of the following commands realize that they are going to delete your files. And as cool as I think they are I have no idea what your setup looks like. So before you run any of them, make sure you have a good, complete, current backup of your data.

Delete files with specific extensions

To remove all the files with a specific file extension, the following examples will set you up. This command will search from the current directory and delete all files with a php5 extension in all subdirectories.

find . -type f -name '*.php5' -delete

Or for clarities sake

find . -type f -name '*.[add your extension here]' -delete

Delete files with a specific filename

If it is a specific filename you are wanting to delete recursively just remove the ‘*.extension’ and replace it with the filename.

find . -type f -name '[insert filename here]' -delete

The following extension will delete all files named “debug.log” recursively through the current folder.

find . -type f -name 'debug.log' -delete

Or maybe you are wanting to delete all the access log files for a nginx from 2018.

find /var/log/nginx/ -type f -name 'access.log-2018*.log' -delete

Usually these commands do what I want. But sometimes I need a command that also has a timeframe

Delete files added within a specific timeframe

After removing all the files with a specific name or extension I decided I needed one more step. So I hunted for a way to find all the files added within the last week. The client discovered the attack just over a week since the last update.

The following command will delete any files beneath the current directory that have been modified within the last 24 hours.

find . -mtime -1 -delete

This command will delete any files beneath the current directory that have been modified within the last 48 hours.

find . -mtime +1 -delete

And this command will delete any files beneath the current directory that have been modified within the last 96 hours. Each additional number adds 24 hours to the check 4, 5, etc. With the first 24 hours implied.

find . -mtime +3 -delete

Combine the two for specific files within a specific timeframe

By combining the commands above you can delete any files ending in .log. Which also sit beneath the current directory that have been modified within the last 96 hours.

find . -mtime +3 -type f -name '*.log' -delete

Those commands saved me a ton of time, and got my client back up and running. Hopefully they can do the same for you.

Force Disconnect iscsi and Reconnect iscsi on centos6+

We’ve all been there, having a webserver which is running low on diskspace. I added an iscsi filesystem to provide sufficient space for creating backups of the system. I host the iscsi filesystem on a FreeNAS box, which usually runs great. But periodically it appears that a bug in the system causes the FreeNAS box to reboot. Then my webservers iscsi connection gets confused and I get to take drastic measures. I force disconnect iscsi and reconnect it.

Force Disconnect iscsi

After ensuring that the FreeNAS box is running properly, I unmount the drive on my webserver.

umount /dev/sda1

I get the details of the existing zombie session with this command.

iscsiadm -m session -o show
 
tcp: [3] 192.168.0.253:3261,2 iqn.2005-10.org.freenas.ctl:tmpfs (non-flash) 

The response shows the details of the active session, the IP address and port, as well as the iqn string. I use them in the following command to force disconnect iscsi.

iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:tmpfs -p 192.168.0.253:3261 -u 
 
Logging out of session [sid: 3, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] 

After getting confirmation that the session has been logged out, I then restart the iscsi service for good measure.

service iscsid restart

Reconnect iscsi

Now that the iscsi service has been restarted the system is ready to reconnect to the iscsi filesystem. Execute the following command to log back in to your iscsi connections.

iscsiadm -m node --login 

Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:tmpfs, portal: 192.168.0.253,3261] successful. 

Once the login has been successful you should be ready to mount your drive again. But I have found that sometimes when reconnecting the filesystem it gives it a new drive identifier. My system will bounce between /dev/sda and /dev/sdb. I use the following command to verify which drive identifier it has used.

Remount your iscsi filesytem

cat /proc/partitions 
 major minor  #blocks  name
 

   11        0    1048575 sr0
  252        0  524288000 vda
  252        1  505411392 vda1
  252        2   18874368 vda2
  252       16  104857600 vdb
  252       17  104857568 vdb1
    8        0 1310720000 sda
    8        1  734002176 sda1 

The last 2 lines show that it connected as /dev/sda where it had been /dev/sdb before it was disconnected. I then change the settings in /etc/fstab to use the new drive identifier. After that you might be good to go, but I have also found when the drive identifier changes that issues with UUID’s can occur. As in my case when I attempted to mount the filesystem.

mount /var/lib/psa/dumps/ 
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so 

That didn’t look very good, so I ran the command it recommended to check dmesg.

dmesg | tail
 [9638158.235890] sd 5:0:0:0: [sda] 2621440000 512-byte logical blocks: (1.34 TB/1.22 TiB)
 [9638158.235895] sd 5:0:0:0: [sda] 32768-byte physical blocks
 [9638158.237537] sd 5:0:0:0: [sda] Write Protect is off
 [9638158.237541] sd 5:0:0:0: [sda] Mode Sense: 7f 00 10 08
 [9638158.238100] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
 [9638158.238193] sd 5:0:0:0: alua: transition timeout set to 60 seconds
 [9638158.238204] sd 5:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
 [9638158.252324]  sda: sda1
 [9638158.256859] sd 5:0:0:0: [sda] Attached SCSI disk
 [9638245.008254] XFS (sda1): Filesystem has duplicate UUID 1615e711-25b3-4120-aa87-2514feebb1c4 - can't mount 

The last line of the output indicated that the filesytem had a duplicate UUID (because it used to be /dev/sdb). XFS tools v5+ have a command to change the UUID of a filesystem. But I was running v3.1.1, so I opted for a different solution. I added an option to the fstab entry that tells the system not to check the UUID.

/dev/sda1 /var/lib/psa/dumps/ xfs _netdev,nouuid 0 0 

Now the filesystem mounts fine, even with a duplicate UUID. This may not be the perfect solution. But when you need to force disconnect iscsi and reconnect iscsi, these commands do the trick for me.

Replacement for Netstat on Linux

Linux distributions grow, get updated, and more advanced. Most tools that you have grown accustomed to using remain, But a few have been replaced. Many reasons for the replacement of core tools. Most of these tools are removed because they are inefficient and replaced with better alternatives. Old networking tools like netstat and ifconfig work by accessing files in the /proc filesystem to aggregate their response. This behavior has been fine for years, especially on smaller systems. As systems grow and complexity increases the chances for inaccurate responses have increased. So what is the replacement for netstat?

The Linux networking subsystem has continued to evolve with the changing of the Linux Kernel, which has lead to new networking functions and processes. The new tools leverage Netlink Sockets to provide their information, which are much more efficient. For a full writeup on the reasons for the switch over I recommend this article on the real reasons of the Linux replacement for netstat.

Netstat is my goto tool for getting a list of open ports on a system and associated processes. Even with the changes in the works I have loaded the program from the net-tools package, until I couldn’t. Recently I was helping a client with a OpenSuse 15 system that didn’t have netstat, and it could not locate the package when I attempted to install it. I then tried the alternative to netstat, ss.

SS is the replacement for netstat

SS stands for “Socket Statistics” and operates in a manner similar to netstat. It formats things differently, leading to some required adjustments. Here is the basic help output for the ss command.

SS Usage

bdoga@webserver:~$ ss --help
 Usage: ss [ OPTIONS ]
        ss [ OPTIONS ] [ FILTER ]
    -h, --help          this message
    -V, --version       output version information
    -n, --numeric       don't resolve service names
    -r, --resolve       resolve host names
    -a, --all           display all sockets
    -l, --listening     display listening sockets
    -o, --options       show timer information
    -e, --extended      show detailed socket information
    -m, --memory        show socket memory usage
    -p, --processes     show process using socket
    -i, --info          show internal TCP information
    -s, --summary       show socket usage summary
    -b, --bpf           show bpf filter socket information
    -E, --events        continually display sockets as they are destroyed
    -Z, --context       display process SELinux security contexts
    -z, --contexts      display process and socket SELinux security contexts
    -N, --net           switch to the specified network namespace name
 

    -4, --ipv4          display only IP version 4 sockets
    -6, --ipv6          display only IP version 6 sockets
    -0, --packet        display PACKET sockets
    -t, --tcp           display only TCP sockets
    -u, --udp           display only UDP sockets
    -d, --dccp          display only DCCP sockets
    -w, --raw           display only RAW sockets
    -x, --unix          display only Unix domain sockets
    -f, --family=FAMILY display sockets of type FAMILY
 

    -A, --query=QUERY, --socket=QUERY
        QUERY := {all|inet|tcp|udp|raw|unix|unix_dgram|unix_stream|unix_seqpacket|packet|netlink}[,QUERY]
 

    -D, --diag=FILE     Dump raw information about TCP sockets to FILE
    -F, --filter=FILE   read filter information from FILE
        FILTER := [ state STATE-FILTER ] [ EXPRESSION ]
        STATE-FILTER := {all|connected|synchronized|bucket|big|TCP-STATES}
          TCP-STATES := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|closed|close-wait|last-ack|listen|closing}
           connected := {established|syn-sent|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
        synchronized := {established|syn-recv|fin-wait-{1,2}|time-wait|close-wait|last-ack|closing}
              bucket := {syn-recv|time-wait}
                 big := {established|syn-sent|fin-wait-{1,2}|closed|close-wait|last-ack|listen|closing} 

So while it may not be netstat it appears to be a very robust tool. Ss allows the querying of specific information and then filter the results directly. It is useful to filter directly without relying on a tool like grep or awk. But there is nothing to stop you from piping the output into the tool of your choice.

SS Usage Examples

bdoga@webserver:~$ ss -lt
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                    
 LISTEN     0      128        *:6032                     *:*                    
 LISTEN     0      128        *:http                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:6033                     *:*                    
 LISTEN     0      128        *:ssh                      *:*                    
 LISTEN     0      128        *:https                    *:*                    
 LISTEN     0      128       :::ssh                     :::*               

Show all the tcp ports “-t” that the server is listening “-l” to.

bdoga@webserver:~$ sudo ss -ltp
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("proxysql_galera",pid=28271,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("proxysql_galera",pid=28271,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=950,fd=4)) 

Show all the tcp ports “-t” that the server is listening “-l” on. What process “-p” are using those ports. The “-p” flag requires root/sudo access to properly display which processes are using those ports.

bdoga@webserver:~$ sudo ss -lup
State      Recv-Q Send-Q Local Address:Port     Peer Address:Port       
 UNCONN     0      0          *:59879                    *:*                     users:(("rsyslogd",pid=556,fd=5))
 UNCONN     0      0      192.168.28.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=21))
 UNCONN     0      0      10.10.10.28:ntp                *:*                     users:(("ntpd",pid=9421,fd=20))
 UNCONN     0      0      209.33.221.34:ntp              *:*                     users:(("ntpd",pid=9421,fd=19))
 UNCONN     0      0      127.0.0.1:ntp                  *:*                     users:(("ntpd",pid=9421,fd=18))
 UNCONN     0      0          *:ntp                      *:*                     users:(("ntpd",pid=9421,fd=17))
 UNCONN     0      0          *:snmp                     *:*                     users:(("snmpd",pid=1045,fd=7))
 UNCONN     0      0          *:https                    *:*                     users:(("litespeed",pid=21324,fd=18),("litespeed",pid=21321,fd=18))
 UNCONN     0      0      127.0.0.1:8822                 *:*                     users:(("Site24x7Agent",pid=1141,fd=19))
 UNCONN     0      0      fe80::218:3eff:fe57:7bf2%eth1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=24))
 UNCONN     0      0      fe80::216:3cff:fe4e:8673%eth0:ntp                     :::*                     users:(("ntpd",pid=9421,fd=23))
 UNCONN     0      0        ::1:ntp                     :::*                     users:(("ntpd",pid=9421,fd=22))
 UNCONN     0      0         :::ntp                     :::*                     users:(("ntpd",pid=9421,fd=16)) 

Same as the previous example but listing udp ports “-u”.

bdoga@webserver:~$ sudo ss -at4p
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port       
 LISTEN     0      128        *:7080                     *:*                     users:(("litespeed",pid=21324,fd=11),("litespeed",pid=21321,fd=11))
 LISTEN     0      128        *:6032                     *:*                     users:(("mysql",pid=28336,fd=28),("timeout",pid=28335,fd=28),("proxysql_galera",pid=28333,fd=28),("proxysql_galera",pid=28332,fd=28),("proxysql_galera",pid=27839,fd=28),("proxysql",pid=3649,fd=28))
 LISTEN     0      128        *:http                     *:*                     users:(("litespeed",pid=21324,fd=9),("litespeed",pid=21321,fd=9))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=22),("timeout",pid=28335,fd=22),("proxysql_galera",pid=28333,fd=22),("proxysql_galera",pid=28332,fd=22),("proxysql_galera",pid=27839,fd=22),("proxysql",pid=3649,fd=22))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=21),("timeout",pid=28335,fd=21),("proxysql_galera",pid=28333,fd=21),("proxysql_galera",pid=28332,fd=21),("proxysql_galera",pid=27839,fd=21),("proxysql",pid=3649,fd=21))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=20),("timeout",pid=28335,fd=20),("proxysql_galera",pid=28333,fd=20),("proxysql_galera",pid=28332,fd=20),("proxysql_galera",pid=27839,fd=20),("proxysql",pid=3649,fd=20))
 LISTEN     0      128        *:6033                     *:*                     users:(("mysql",pid=28336,fd=19),("timeout",pid=28335,fd=19),("proxysql_galera",pid=28333,fd=19),("proxysql_galera",pid=28332,fd=19),("proxysql_galera",pid=27839,fd=19),("proxysql",pid=3649,fd=19))
 LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=950,fd=3))
 LISTEN     0      128        *:https                    *:*                     users:(("litespeed",pid=21324,fd=10),("litespeed",pid=21321,fd=10))
 TIME-WAIT  0      0      209.33.221.34:58184                209.33.221.83:mysql                
 TIME-WAIT  0      0      127.0.0.1:34804             127.0.0.1:6032                 
 TIME-WAIT  0      0      127.0.0.1:34470             127.0.0.1:6032                 
 TIME-WAIT  0      0      209.33.221.34:45368         209.33.221.36:mysql                
 ESTAB      0      0      209.33.221.34:37194         209.33.221.67:mysql                 users:(("mysql",pid=28336,fd=109),("timeout",pid=28335,fd=109),("proxysql_galera",pid=28333,fd=109),("proxysql_galera",pid=28332,fd=109),("proxysql_galera",pid=27839,fd=109),("proxysql",pid=3649,fd=109))
 TIME-WAIT  0      0      127.0.0.1:33812                127.0.0.1:6032                  

To show all “-a” tcp “-t” but only IPv4 “-4” interfaces and their associated processes “-p”.

I will take a little while to fully make the switch since I have used netstat for decades. Although it appears the future is bright, and ss should be able to take care of me in the future. Hopefully this overview helps you make the switch as well.

How to copy a file into all subdirectories

I was left with a conundrum, I had needed to add a default index.html file into each subdirectory on my web server to ensure that the appropriate response was given if someone browsed to any directory on the web server. The command I came up with to copy the file into all subdirectories was:

ls -d */ | xargs -n 1 cp -i index.html

Where index.html is the file to be copied. The ls -d */ command gets a list of directories in the current directory that can be piped into xargs to execute the copying command, but it wasn’t as robust as I was hoping. This command doesn’t work with directories that have spaces in the name. And it only copies the file into the immediate subdirectories. So I started playing around with the command and came up with the following modified command that will copy a file into all subdirectories recursively.

ls -R | grep ":" | sed "s/^.://" | sed "s/://" | xargs -n 1 cp -n index.html 

Again this command is copying the index.html file, ls -R gets a list of all files and directories recursively from the current folder. grep “:” locates all the directories since they each end with a colon “:”. sed “s/^.://” removes the reference to the current directory “.” in the returned directory list. sed “s/^.://” then cleans off the trailing colons “:” from each directory entry and then the cleaned directories are piped into the xargs command to copy the file into each one.

So give it a go, it could save you a bit of time and hassle now that you can copy a file into all subdirectories.

Reinstall Grub after using CloneZilla

I had made a partition backup of a machine using Clonezilla and wanted to restore it. The restore was successful but because I had only restored the partitions rather than the full disk Grub was not installed in the MBR. Without Grub in the MBR the system failed to boot.

I mounted the new filesystem to /mnt while still using the live Clonezilla disk that I had used for the backup. Then I chrooted using the following command

chroot /mnt

then while in the chroot I attempted to reinstall grub, and since this was a Cloudlinux/Centos install I performed

grub-install /dev/sda

But grub-install complained that it couldn’t find /dev/sda or that /dev/sda was not a valid block device. So then hunting around on the internet for a little bit I came across this article which showed basically how to ensure that your current live

  • /dev
  • /sys
  • /proc

filesystems are accessible inside of your current chroot.

So I ran the following commands outside of the chroot before entering it again.

mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys

Then I chrooted to /mnt again and ran my grub-install command and all was well. The machine booted perfectly after that.