How To Identify The Bad Processes On A Hacked Linux Box

Want to help support this blog? Try out Oh Dear, the best all-in-one monitoring tool for your entire website, co-founded by me (the guy that wrote this blogpost). Start with a 10-day trial, no strings attached.

We offer uptime monitoring, SSL checks, broken links checking, performance & cronjob monitoring, branded status pages & so much more. Try us out today!

Profile image of Mattias Geniar

Mattias Geniar, August 08, 2008

Follow me on Twitter as @mattiasgeniar

Let's say you have a hacked Linux server. Let's say you want to find out who did it, as well as what the server is doing now. It's not always easy, but this guide will show you how to use tools like netstat and lsof to find which processes are running on which ports.

First of all, if you still have SSH access to the server, log in through a remote session. Otherwise, you’ll need physical access to the server.

Most hacked servers have only 2 primary functions: try to hack other servers, in order to create a bigger botnet, or to wreck havoc on an unsuspecting server. They do have one thing in common: several outbound connections are made to other servers (portscan, irc-bot, ftp server, …). We’ll identify the ports used for those outbound connections, and the user/process that is executing them. Then we’ll locate the files for those processes, and disable them (after analyzing them).

Note; this article turned out a bit larger than I had thought, so prepare for a wall of text, with the occasional comic – to keep you entertained ;-)

Holy jokers Batman, how do we do that?!

So at first, a little warning. If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ps, ls, lsof, netstat, …) have been replaced by hacked versions, meaning they won’t show you the full information.

Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.

First we need to find out what the server is doing. Seeing as a hacked server is pretty useless if it isn’t making contact with other servers, we can do this by checking all our incoming/outgoing connections. This can help us find out which ports are being used, and what the server’s doing.

To get a list of all current active connections, you can use the netstat command.

$ netstat -an

With the -a parameter, we tell the command to show all connections (including the LISTENING ones), instead of only those who are connected.

The -n parameter shows the different port numbers used. By default, it would display items with a textual description (such as :pop3, :smtp, :http). It’s more useful when we can see the actual ports, shown as :110, :25, :80.

This will generate a list, much like this one (if it’s a standard webserver, it could differ for your server(s) ). I’ve shortened it a bit, it’s usually a lot longer.

$ netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address     Foreign Address     State
tcp        0      0 0.0.0.0:110       0.0.0.0:*           LISTEN     
tcp        0      0 0.0.0.0:143       0.0.0.0:*           LISTEN     
tcp        0      0 0.0.0.0:21        0.0.0.0:*           LISTEN          
tcp        0      0 0.0.0.0:25        0.0.0.0:*           LISTEN   
tcp        0      0 :::80             :::*                LISTEN     
tcp        0      0 :::22             :::*                LISTEN     
tcp        0      0 :::443            :::*                LISTEN     

Active UNIX domain sockets (servers and established)
Proto Flags     Type      State        I-Node Path
unix  [ ACC ]   STREAM    LISTENING    548928452 /var/lib/mysql/mysql.sock
unix  [ ]       DGRAM                  548927555 @/org/kernel/udev/udevd
unix  [ ACC ]   STREAM    LISTENING    548928520 /tmp/spamd_full.sock
unix  [ ]       DGRAM                  548927910 /dev/log
unix  [ ACC ]   STREAM    LISTENING    548928786 /var/run/saslauthd/mux
unix  [ ]       DGRAM                  3285180947
unix  [ ]       DGRAM                  550618919
unix  [ ]       DGRAM                  550618862
unix  [ ]       DGRAM                  548928785
unix  [ ]       DGRAM                  548928764
unix  [ ]       STREAM    CONNECTED    548928535
unix  [ ]       STREAM    CONNECTED    548928534

You’ll probably notice that these connections all look harmless. There are processes listening on port 80 (HTTP), 110 (POP3), 21 (FTP), 25 (SMTP), 22 (SSH) and 443 (HTTPS). For this example, we’ll assume we want to find out which process is listening on port 80.

Note; if your server is hacked and being used as an IRC Bot you’ll see many connections in the range of 6660-7000. You could follow the same steps, by just replacing port 80 by port 6660. It’s just easier to show it this way.

XKCD - Hacked, SQL Injection

Now, to find out which user and which process is making connections on this port, we’ll use the lsof (List Open Files) command.

$ lsof -i tcp:80 -P -R

This command shows us all running processes that are using port 80 for any kind of communication.

  • The -i parameter specifies we want to list the processes, by identifying them with IPv4 or IPv6.
  • The tcp:80 part means we only want to show TCP connections (and ignore UDP for the time being) using port 80.
  • Using -P we specify we want to see the port numbers (80, 21, …) instead of the names (HTTP, FTP) which are shown by default. Much like the -n parameter with netstat.
  • With -R we also show the Parent Process ID, to see who initiated this process.

It will show output like this.

$ lsof -i tcp:80 -P -R
COMMAND   PID  PPID   USER   FD   TYPE    DEVICE SIZE NODE NAME
httpd   13538 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   14279 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   16324     1   root    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   28042 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   29772 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   32424 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)

Wow! Lots of numbers!

Take a deep breath. It looks more complicated than it is, trust me. What can we learn from this? We see 6 instances of httpd, the HTTP daemon that is acting as the webserver. Five of these processes are run by the apache user, and one by root.

All apache httpd processes were started by a main process (PID 16324) which was executed by the root user (PPID column).

The column “command” displays the name of the process. Under “PID” we can see the Process ID, this can be used to easily stop, or kill the process. With “PPID” we can see the process ID that started this specific process.

Beneath “USER” we see the user that is executing the process. We can now see that all httpd processes that are being run as the apache user, were started by Parent Process ID 16324, the httpd process executed as root.

Note; you should be alerted when you see a postgres user running all sorts of programs on all sorts of exotic ports. If that’s the case, the server’s probably hacked, and used for means it wasn’t supposed to.

Now you have a list of all the processes that are using a specific port to communicate. Let’s find out which files these processes use, where they are located, and then stop the process – to prevent further harm. We won’t delete the files that were used just yet, because we might learn something afterwards when we examine them. We will, however, move them outside the user’s directory.

$ lsof -p 16324

With -p we can specify a Process ID (PID) to filter our results to. This’ll only list open files for the PID 16324.

$ lsof -p 16324
COMMAND   PID USER   FD   TYPE    DEVICE    SIZE       NODE NAME
httpd   16324 root  cwd    DIR      0,61    4096   14624004 /
httpd   16324 root  rtd    DIR      0,61    4096   14624004 /
httpd   16324 root  txt    REG      0,61  312372   19768930 /usr/sbin/httpd
httpd   16324 root  mem    REG      0,61   11704   20455174 /usr/lib/libgpg-error.so.0.3.0
httpd   16324 root  mem    REG      0,61  346148   20455190 /usr/lib/libgcrypt.so.11.2.2
httpd   16324 root  mem    REG      0,61  213116   20455786 /usr/lib/libxslt.so.1.1.17
httpd   16324 root  mem    REG      0,61   67932   20455784 /usr/lib/libexslt.so.0.8.13
httpd   16324 root  mem    REG      0,61   23172   20513414 /usr/lib/php/modules/xsl.so
httpd   16324 root  mem    REG      0,61   32216   20513412 /usr/lib/php/modules/xmlwriter.so</span>
....

This list usually goes on for a while, as any given process could have hundreds of files open. However, this list will tell you where the culprit is located. In our case, the program started in /usr/sbin/httpd (see line 3).

Let’s end the process using kill, remove the execute permissions, and move the files to another location.

$ kill 16324

The kill command ends a process with a given PID, in this case 16324.

$ chmod -x /usr/sbin/httpd

The chmod command lets you edit the permissions to this file. We’ll use it to remove any execution rights it might have “-x”.

$ mkdir /home/tmp-sandbox/
 mv /usr/sbin/httpd /home/tmp-sandbox/

We’ll create the directory “/home/tmp-sandbox/” and copy the files there, so we can view them later without risk.

Ok Batman, I stopped the program. Can I go out and play now?

Oh no, not yet.

You may have ended the program, but you still have to prevent if from being started again. First of all, check to see if the program was filed under /etc/init.d . Every command located in this directory will be executed upon boot – and you don’t want the process to restart if you have to reboot the server.

Next, make sure it’s not created a crontab entry to be executed every X-seconds.

$ crontab -l

This will list all your cronjobs, for the current user – probably root. If you see any command in there that is started from the location where you just found your culprit, remove it from the crontab.

$ crontab -e

This lets you edit the cronjobs, just like a normal vi. Remove the line, and save the file (:wq). Also make sure to log in to the user that executed your bad process (su ) and check those cronjobs.

Next up, update all your software. Either via yum or apt-get, or manually – just make sure it’s all up to date. Also, avoid using tools such as Webmin to manage a server – it’s highly vulnerable to these kinds of attacks.

You might want to check your /etc/, /tmp/ and /home/ folders just in case, for any files that were left there by your hacker. Remember: it’s not because you stopped the process, that you’re safe.

Assume there are other files, still hidden, on your server. Find them, and disable them. Then study their contents, and learn from it.

If this article was in any way useful to you, please leave a message and state how it helped you. If you see errors, of have other tips – don’t hesitate to share!



Want to subscribe to the cron.weekly newsletter?

I write a weekly-ish newsletter on Linux, open source & webdevelopment called cron.weekly.

It features the latest news, guides & tutorials and new open source projects. You can sign up via email below.

No spam. Just some good, practical Linux & open source content.