How To Identify The Bad Processes On A Hacked Linux Box

Mattias Geniar, Friday, August 8, 2008 - last modified: Saturday, August 8, 2015

Let's say you have a hacked Linux server. Let's say you want to find out who did it, as well as what the server is doing now. It's not always easy, but this guide will show you how to use tools like netstat and lsof to find which processes are running on which ports.

First of all, if you still have SSH access to the server, log in through a remote session. Otherwise, you'll need physical access to the server.

Most hacked servers have only 2 primary functions: try to hack other servers, in order to create a bigger botnet, or to wreck havoc on an unsuspecting server. They do have one thing in common: several outbound connections are made to other servers (portscan, irc-bot, ftp server, ...). We'll identify the ports used for those outbound connections, and the user/process that is executing them. Then we'll locate the files for those processes, and disable them (after analyzing them).

Note; this article turned out a bit larger than I had thought, so prepare for a wall of text, with the occasional comic -- to keep you entertained ;-)

Holy jokers Batman, how do we do that?!

So at first, a little warning. If you're trying to fix a hacked server, it's very possible that the binaries you're executing (such as ps, ls, lsof, netstat, ...) have been replaced by hacked versions, meaning they won't show you the full information.

Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.

First we need to find out what the server is doing. Seeing as a hacked server is pretty useless if it isn't making contact with other servers, we can do this by checking all our incoming/outgoing connections. This can help us find out which ports are being used, and what the server's doing.

To get a list of all current active connections, you can use the netstat command.

$ netstat -an

With the -a parameter, we tell the command to show all connections (including the LISTENING ones), instead of only those who are connected.

The -n parameter shows the different port numbers used. By default, it would display items with a textual description (such as :pop3, :smtp, :http). It's more useful when we can see the actual ports, shown as :110, :25, :80.

This will generate a list, much like this one (if it's a standard webserver, it could differ for your server(s) ). I've shortened it a bit, it's usually a lot longer.

$ netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address     Foreign Address     State
tcp        0      0 0.0.0.0:110       0.0.0.0:*           LISTEN     
tcp        0      0 0.0.0.0:143       0.0.0.0:*           LISTEN     
tcp        0      0 0.0.0.0:21        0.0.0.0:*           LISTEN          
tcp        0      0 0.0.0.0:25        0.0.0.0:*           LISTEN   
tcp        0      0 :::80             :::*                LISTEN     
tcp        0      0 :::22             :::*                LISTEN     
tcp        0      0 :::443            :::*                LISTEN     

Active UNIX domain sockets (servers and established)
Proto Flags     Type      State        I-Node Path
unix  [ ACC ]   STREAM    LISTENING    548928452 /var/lib/mysql/mysql.sock
unix  [ ]       DGRAM                  548927555 @/org/kernel/udev/udevd
unix  [ ACC ]   STREAM    LISTENING    548928520 /tmp/spamd_full.sock
unix  [ ]       DGRAM                  548927910 /dev/log
unix  [ ACC ]   STREAM    LISTENING    548928786 /var/run/saslauthd/mux
unix  [ ]       DGRAM                  3285180947
unix  [ ]       DGRAM                  550618919
unix  [ ]       DGRAM                  550618862
unix  [ ]       DGRAM                  548928785
unix  [ ]       DGRAM                  548928764
unix  [ ]       STREAM    CONNECTED    548928535
unix  [ ]       STREAM    CONNECTED    548928534

You'll probably notice that these connections all look harmless. There are processes listening on port 80 (HTTP), 110 (POP3), 21 (FTP), 25 (SMTP), 22 (SSH) and 443 (HTTPS). For this example, we'll assume we want to find out which process is listening on port 80.

Note; if your server is hacked and being used as an IRC Bot you'll see many connections in the range of 6660-7000. You could follow the same steps, by just replacing port 80 by port 6660. It's just easier to show it this way.

XKCD - Hacked, SQL Injection

Now, to find out which user and which process is making connections on this port, we'll use the lsof (List Open Files) command.

$ lsof -i tcp:80 -P -R

This command shows us all running processes that are using port 80 for any kind of communication.

  • The -i parameter specifies we want to list the processes, by identifying them with IPv4 or IPv6.
  • The tcp:80 part means we only want to show TCP connections (and ignore UDP for the time being) using port 80.
  • Using -P we specify we want to see the port numbers (80, 21, ...) instead of the names (HTTP, FTP) which are shown by default. Much like the -n parameter with netstat.
  • With -R we also show the Parent Process ID, to see who initiated this process.

It will show output like this.

$ lsof -i tcp:80 -P -R
COMMAND   PID  PPID   USER   FD   TYPE    DEVICE SIZE NODE NAME
httpd   13538 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   14279 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   16324     1   root    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   28042 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   29772 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   32424 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)

Wow! Lots of numbers!

Take a deep breath. It looks more complicated than it is, trust me. What can we learn from this? We see 6 instances of httpd, the HTTP daemon that is acting as the webserver. Five of these processes are run by the apache user, and one by root.

All apache httpd processes were started by a main process (PID 16324) which was executed by the root user (PPID column).

The column "command" displays the name of the process. Under "PID" we can see the Process ID, this can be used to easily stop, or kill the process. With "PPID" we can see the process ID that started this specific process.

Beneath "USER" we see the user that is executing the process. We can now see that all httpd processes that are being run as the apache user, were started by Parent Process ID 16324, the httpd process executed as root.

Note; you should be alerted when you see a postgres user running all sorts of programs on all sorts of exotic ports. If that's the case, the server's probably hacked, and used for means it wasn't supposed to.

Now you have a list of all the processes that are using a specific port to communicate. Let's find out which files these processes use, where they are located, and then stop the process -- to prevent further harm. We won't delete the files that were used just yet, because we might learn something afterwards when we examine them. We will, however, move them outside the user's directory.

$ lsof -p 16324

With -p we can specify a Process ID (PID) to filter our results to. This'll only list open files for the PID 16324.

$ lsof -p 16324
COMMAND   PID USER   FD   TYPE    DEVICE    SIZE       NODE NAME
httpd   16324 root  cwd    DIR      0,61    4096   14624004 /
httpd   16324 root  rtd    DIR      0,61    4096   14624004 /
httpd   16324 root  txt    REG      0,61  312372   19768930 /usr/sbin/httpd
httpd   16324 root  mem    REG      0,61   11704   20455174 /usr/lib/libgpg-error.so.0.3.0
httpd   16324 root  mem    REG      0,61  346148   20455190 /usr/lib/libgcrypt.so.11.2.2
httpd   16324 root  mem    REG      0,61  213116   20455786 /usr/lib/libxslt.so.1.1.17
httpd   16324 root  mem    REG      0,61   67932   20455784 /usr/lib/libexslt.so.0.8.13
httpd   16324 root  mem    REG      0,61   23172   20513414 /usr/lib/php/modules/xsl.so
httpd   16324 root  mem    REG      0,61   32216   20513412 /usr/lib/php/modules/xmlwriter.so
....

This list usually goes on for a while, as any given process could have hundreds of files open. However, this list will tell you where the culprit is located. In our case, the program started in /usr/sbin/httpd (see line 3).

Let's end the process using kill, remove the execute permissions, and move the files to another location.

$ kill 16324

The kill command ends a process with a given PID, in this case 16324.

$ chmod -x /usr/sbin/httpd

The chmod command lets you edit the permissions to this file. We'll use it to remove any execution rights it might have "-x".

$ mkdir /home/tmp-sandbox/
 mv /usr/sbin/httpd /home/tmp-sandbox/

We'll create the directory "/home/tmp-sandbox/" and copy the files there, so we can view them later without risk.

Ok Batman, I stopped the program. Can I go out and play now?

Oh no, not yet.

You may have ended the program, but you still have to prevent if from being started again. First of all, check to see if the program was filed under /etc/init.d . Every command located in this directory will be executed upon boot -- and you don't want the process to restart if you have to reboot the server.

Next, make sure it's not created a crontab entry to be executed every X-seconds.

$ crontab -l

This will list all your cronjobs, for the current user -- probably root. If you see any command in there that is started from the location where you just found your culprit, remove it from the crontab.

$ crontab -e

This lets you edit the cronjobs, just like a normal vi. Remove the line, and save the file (:wq). Also make sure to log in to the user that executed your bad process (su <user>) and check those cronjobs.

Next up, update all your software. Either via yum or apt-get, or manually -- just make sure it's all up to date. Also, avoid using tools such as Webmin to manage a server -- it's highly vulnerable to these kinds of attacks.

You might want to check your /etc/, /tmp/ and /home/ folders just in case, for any files that were left there by your hacker. Remember: it's not because you stopped the process, that you're safe.

Assume there are other files, still hidden, on your server. Find them, and disable them. Then study their contents, and learn from it.

If this article was in any way useful to you, please leave a message and state how it helped you. If you see errors, of have other tips -- don't hesitate to share!



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Rafal Los Saturday, August 9, 2008 at 06:57 - Reply

You appear to assume that the box’s processes and files haven’t been replaced with trojaned versions to hide the original malware…


Mattias Geniar Saturday, August 9, 2008 at 10:48 - Reply

Well, this is indeed the simplified version. Most hacked servers are done using simple techniques, to do simple things, which can then simply be discovered.

Perhaps I’ll expand this to identify trojans and other malware too, for now it’ll just be this basic version.


Pete Monday, September 22, 2008 at 15:03 - Reply

This was a very helpful little tutorial on the practical use of lsof. Thanks for taking the time to write this!

–Pete


Tim Tuesday, April 7, 2009 at 16:36 - Reply

Great article! Very helpful. Much appreciated!


MikeT Thursday, May 6, 2010 at 19:03 - Reply

This really opened my eyes. Thanks for using your time to write this…

MikeT


Emre Sevinç Friday, May 21, 2010 at 12:57 - Reply

You said: “Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.”

Booting the hacked server with a clean BackTrack Live CD and then analyzing the connections? How can that happen? I mean if you boot it with a clean, live CD then it is a clean, live server and no suspicious connections exist because you just booted it with a live CD. Your hacked server is not booted anymore. I’m a little bit confused.


Matti Sunday, May 23, 2010 at 10:06 - Reply

@Emre; it only goes back to the “If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ‘ps’, ‘ls’, lsof’, ‘netstat’, …) have been replaced by hacked versions”-line. Those binaries shouldn’t be trusted, and can be “replaced” with the live CD ones.

The open connections are indeed useless at that point.


zobin Tuesday, October 26, 2010 at 18:48 - Reply

Thank you for your article,and i found the spam process,but i don’t how to clear it in one time.
The below is the list which i ran lsof -p 2628(the pid of the main spam process)

COMMAND   PID USER   FD   TYPE     DEVICE    SIZE    NODE NAME
sendmail 2628 root  cwd    DIR        8,2 1699840 3899771 /var/spool/mqueue
sendmail 2628 root  rtd    DIR        8,2    4096       2 /
sendmail 2628 root  txt    REG        8,2  806460 6567022 /usr/sbin/sendmail.sendmail
sendmail 2628 root  mem    REG        8,2         7733281 /lib/libnsl-2.5.so (path inode=7733286)
sendmail 2628 root  mem    REG        8,2         7733338 /lib/libwrap.so.0.7.6 (path inode=7733452)
sendmail 2628 root  mem    REG        8,2         6563225 /usr/lib/libhesiod.so.0.0.0 (path inode=6563806)
sendmail 2628 root  mem    REG        8,2         7733446 /lib/libcom_err.so.2.1 (path inode=7734509)
sendmail 2628 root  mem    REG        8,2   14848 6654557 /usr/lib/sasl2/libplain.so.2.0.22
sendmail 2628 root  mem    REG        8,2         7733275 /lib/libcrypt-2.5.so (path inode=7734506)
sendmail 2628 root  DEL    REG        8,2         6565112 /usr/lib/liblber-2.3.so.0.2.31.#prelink#.oxts6J
sendmail 2628 root  mem    REG        8,2         6565055 /usr/lib/libkrb5.so.3.3 (path inode=6573533)
sendmail 2628 root  DEL    REG        8,2         6565057 /usr/lib/libkrb5support.so.0.1.#prelink#.QePuRn
sendmail 2628 root  mem    REG        8,2         6565045 /usr/lib/libk5crypto.so.3.1 (path inode=6573532)
sendmail 2628 root  mem    REG        8,2         6561754 /usr/lib/libz.so.1.2.3 (path inode=6567559)
sendmail 2628 root  mem    REG        8,2         7733295 /lib/libpthread-2.5.so (path inode=7733278)
sendmail 2628 root  mem    REG        8,2         7733264 /lib/ld-2.5.so (path inode=7733265)
sendmail 2628 root  mem    REG        8,2         7733339 /lib/libkeyutils-1.2.so (path inode=7733472)
sendmail 2628 root  mem    REG        8,2         7733332 /lib/libsepol.so.1 (path inode=7733300)
sendmail 2628 root  DEL    REG        8,2         7733467 /lib/libcrypto.so.0.9.8e.#prelink#.SBMg0S
sendmail 2628 root  mem    REG        8,2         7733277 /lib/libdl-2.5.so (path inode=7733296)
sendmail 2628 root  DEL    REG        8,2         6565114 /usr/lib/libldap-2.3.so.0.2.31.#prelink#.093P7r
sendmail 2628 root  mem    REG        8,2         7733297 /lib/libresolv-2.5.so (path inode=7734508)
sendmail 2628 root  mem    REG        8,2         7733469 /lib/libssl.so.0.9.8e (path inode=7734511)
sendmail 2628 root  mem    REG        8,2  905200 6654584 /usr/lib/sasl2/libsasldb.so.2.0.22
sendmail 2628 root  mem    REG        8,2         7733462 /lib/libdb-4.3.so (path inode=7733448)
sendmail 2628 root  mem    REG        8,2         7733271 /lib/libc-2.5.so (path inode=7733272)
sendmail 2628 root  mem    REG        8,2   50848 7733287 /lib/libnss_files-2.5.so
sendmail 2628 root  DEL    REG        8,2         6565041 /usr/lib/libgssapi_krb5.so.2.2.#prelink#.GnGQSz
sendmail 2628 root  mem    REG        8,2         7733442 /lib/libselinux.so.1 (path inode=7733304)
sendmail 2628 root  mem    REG        8,2   14752 6654553 /usr/lib/sasl2/liblogin.so.2.0.22
sendmail 2628 root  mem    REG        8,2         6562037 /usr/lib/libsasl2.so.2.0.22 (path inode=6563124)
sendmail 2628 root  mem    REG        8,2   14372 6652653 /usr/lib/sasl2/libanonymous.so.2.0.22
sendmail 2628 root    0r   CHR        1,3            1290 /dev/null
sendmail 2628 root    1w   CHR        1,3            1290 /dev/null
sendmail 2628 root    2w   CHR        1,3            1290 /dev/null
sendmail 2628 root    3u  unix 0xc3616580            7786 socket
sendmail 2628 root    4u  IPv4       7787             TCP localhost.localdomain:smtp (LISTEN)
sendmail 2628 root    5wW  REG        8,2      33 3900362 /var/run/sendmail.pid

Mattias Geniar Tuesday, October 26, 2010 at 23:25 - Reply

For starters, stop sendmail so it no longer sends out messages. Then run some kind of rootkit hunter, and parse your log messages.
Sorry I can’t be more precise, but this differs from each server, and the way the server is being abused to send out spam. If possible, contact your hosting provider/sysadmin for more guidance.


julian Wednesday, February 9, 2011 at 23:43 - Reply

Hi Matti,

you said:
@Emre; it only goes back to the “If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ‘ps’, ‘ls’, lsof’, ‘netstat’, …) have been replaced by hacked versions”-line. Those binaries shouldn’t be trusted, and can be “replaced” with the live CD ones.

I am ok with this idea if the replaced binaries are static but if they aren’t how could you trust the librairies used by those binaries ? I think you can’t turst a hacked server until you re-install it… you never now what they did with your server and how much this hacked machine could be corrupted.


Caiya Monday, November 14, 2011 at 16:16 - Reply

Wait, I cannot fathom it being so srtaigthfowrrad.


Rashantha Tuesday, April 4, 2017 at 23:32 - Reply

Thanks. An article on how to catch these guys would be useful.


Hasa Wednesday, April 12, 2017 at 23:40 - Reply

I am sorry if i am asking for a foolish question but being a newbie i need to konw how you realize that the process given below was the culprit as you mentioned…
—————–
This list usually goes on for a while, as any given process could have hundreds of files open. However, this list will tell you where the culprit is located. In our case, the program started in /usr/sbin/httpd (see line 3).
—————–
you are asking about this process here
httpd 16324 root txt REG 0,61 312372 19768930 /usr/sbin/httpd

what i got from this is that any process ran under https is culprit?


joe Tuesday, May 22, 2018 at 22:42 - Reply

How did you know the culprit file as this one?
httpd 16324 root txt REG 0,61 312372 19768930 /usr/sbin/httpd

BTW, my site mentioned is in maintenance mode


Ali Bal Monday, July 9, 2018 at 23:31 - Reply

You may open server with clean boot cd than install clamav and scan & Clean all infected files with clamscan -r –remove / command.

To check files in the USER home directory and remove infected files (WARNING: Files are gone.):

clamscan -r –remove /home/USER

Installing

ClamAV can be found for Ubuntu in the apt repository. Run this command to install ClamAV:

apt-get install clamav

If you need clamd, you may also want to run:

apt-get install clamav-daemon

If you require support for scanning compressed RAR files you first need to enable the non-free archive, and then you can install the RAR-plugin using:

apt-get install libclamunrar6

To update the virus database, open a terminal and enter the following command:

sudo freshclam
To automate this update you can set up a cron job. I’ll show how to update the virus database every day at 8:57 AM. First, open the terminal and su to root.

sudo su
Now you need to modify the crontab for the root user.

crontab -e
This opens the root crontab file in the nano text editor. Add the following line

57 08 * * * freshclam

Presently, the clamav-freshclam service takes care of updating the definitions (as of 15.04 for sure, possibly as early as 12.04). For Nautilus integration, install clamtk-nautilus (and restart nautilus) – michael_n Dec 16 ’15 at 7:08

The right way to go is to edit freshclam.conf and add the number of dayly Checks you want

vim /etc/clamav/freshclam.conf
Checks 24
##Notify clamd to reload it self
NotifyClamd /etc/clamav/clamd.conf


Dwayne Thursday, February 21, 2019 at 07:40 - Reply

Thanks very much, this was exactly what I needed to quickly track down the source of a malware-infected website on one of our server.


Leave a Reply

Your email address will not be published. Required fields are marked *

Inbound links