Let's say you have a hacked Linux server. Let's say you want to find out who did it, as well as what the server is doing now. It's not always easy, but this guide will show you how to use tools like netstat and lsof to find which processes are running on which ports.
First of all, if you still have SSH access to the server, log in through a remote session. Otherwise, you’ll need physical access to the server.
Most hacked servers have only 2 primary functions: try to hack other servers, in order to create a bigger botnet, or to wreck havoc on an unsuspecting server. They do have one thing in common: several outbound connections are made to other servers (portscan, irc-bot, ftp server, …). We’ll identify the ports used for those outbound connections, and the user/process that is executing them. Then we’ll locate the files for those processes, and disable them (after analyzing them).
Note; this article turned out a bit larger than I had thought, so prepare for a wall of text, with the occasional comic – to keep you entertained ;-)
Holy jokers Batman, how do we do that?!
So at first, a little warning. If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ps, ls, lsof, netstat, …) have been replaced by hacked versions, meaning they won’t show you the full information.
Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.
First we need to find out what the server is doing. Seeing as a hacked server is pretty useless if it isn’t making contact with other servers, we can do this by checking all our incoming/outgoing connections. This can help us find out which ports are being used, and what the server’s doing.
To get a list of all current active connections, you can use the netstat command.
$ netstat -an
With the -a parameter, we tell the command to show all connections (including the LISTENING ones), instead of only those who are connected.
The -n parameter shows the different port numbers used. By default, it would display items with a textual description (such as :pop3, :smtp, :http). It’s more useful when we can see the actual ports, shown as :110, :25, :80.
This will generate a list, much like this one (if it’s a standard webserver, it could differ for your server(s) ). I’ve shortened it a bit, it’s usually a lot longer.
$ netstat -an Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 0 :::80 :::* LISTEN tcp 0 0 :::22 :::* LISTEN tcp 0 0 :::443 :::* LISTEN Active UNIX domain sockets (servers and established) Proto Flags Type State I-Node Path unix [ ACC ] STREAM LISTENING 548928452 /var/lib/mysql/mysql.sock unix [ ] DGRAM 548927555 @/org/kernel/udev/udevd unix [ ACC ] STREAM LISTENING 548928520 /tmp/spamd_full.sock unix [ ] DGRAM 548927910 /dev/log unix [ ACC ] STREAM LISTENING 548928786 /var/run/saslauthd/mux unix [ ] DGRAM 3285180947 unix [ ] DGRAM 550618919 unix [ ] DGRAM 550618862 unix [ ] DGRAM 548928785 unix [ ] DGRAM 548928764 unix [ ] STREAM CONNECTED 548928535 unix [ ] STREAM CONNECTED 548928534
You’ll probably notice that these connections all look harmless. There are processes listening on port 80 (HTTP), 110 (POP3), 21 (FTP), 25 (SMTP), 22 (SSH) and 443 (HTTPS). For this example, we’ll assume we want to find out which process is listening on port 80.
Note; if your server is hacked and being used as an IRC Bot you’ll see many connections in the range of 6660-7000. You could follow the same steps, by just replacing port 80 by port 6660. It’s just easier to show it this way.
Now, to find out which user and which process is making connections on this port, we’ll use the lsof (List Open Files) command.
$ lsof -i tcp:80 -P -R
This command shows us all running processes that are using port 80 for any kind of communication.
- The
-iparameter specifies we want to list the processes, by identifying them with IPv4 or IPv6. - The
tcp:80part means we only want to show TCP connections (and ignore UDP for the time being) using port 80. - Using
-Pwe specify we want to see the port numbers (80, 21, …) instead of the names (HTTP, FTP) which are shown by default. Much like the-nparameter with netstat. - With
-Rwe also show the Parent Process ID, to see who initiated this process.
It will show output like this.
$ lsof -i tcp:80 -P -R COMMAND PID PPID USER FD TYPE DEVICE SIZE NODE NAME httpd 13538 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN) httpd 14279 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN) httpd 16324 1 root 4u IPv6 548928212 TCP *:80 (LISTEN) httpd 28042 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN) httpd 29772 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN) httpd 32424 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN)
Wow! Lots of numbers!
Take a deep breath. It looks more complicated than it is, trust me. What can we learn from this? We see 6 instances of httpd, the HTTP daemon that is acting as the webserver. Five of these processes are run by the apache user, and one by root.
All apache httpd processes were started by a main process (PID 16324) which was executed by the root user (PPID column).
The column “command” displays the name of the process. Under “PID” we can see the Process ID, this can be used to easily stop, or kill the process. With “PPID” we can see the process ID that started this specific process.
Beneath “USER” we see the user that is executing the process. We can now see that all httpd processes that are being run as the apache user, were started by Parent Process ID 16324, the httpd process executed as root.
Note; you should be alerted when you see a postgres user running all sorts of programs on all sorts of exotic ports. If that’s the case, the server’s probably hacked, and used for means it wasn’t supposed to.
Now you have a list of all the processes that are using a specific port to communicate. Let’s find out which files these processes use, where they are located, and then stop the process – to prevent further harm. We won’t delete the files that were used just yet, because we might learn something afterwards when we examine them. We will, however, move them outside the user’s directory.
$ lsof -p 16324
With -p we can specify a Process ID (PID) to filter our results to. This’ll only list open files for the PID 16324.
$ lsof -p 16324 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME httpd 16324 root cwd DIR 0,61 4096 14624004 / httpd 16324 root rtd DIR 0,61 4096 14624004 / httpd 16324 root txt REG 0,61 312372 19768930 /usr/sbin/httpd httpd 16324 root mem REG 0,61 11704 20455174 /usr/lib/libgpg-error.so.0.3.0 httpd 16324 root mem REG 0,61 346148 20455190 /usr/lib/libgcrypt.so.11.2.2 httpd 16324 root mem REG 0,61 213116 20455786 /usr/lib/libxslt.so.1.1.17 httpd 16324 root mem REG 0,61 67932 20455784 /usr/lib/libexslt.so.0.8.13 httpd 16324 root mem REG 0,61 23172 20513414 /usr/lib/php/modules/xsl.so httpd 16324 root mem REG 0,61 32216 20513412 /usr/lib/php/modules/xmlwriter.so</span> ....
This list usually goes on for a while, as any given process could have hundreds of files open. However, this list will tell you where the culprit is located. In our case, the program started in /usr/sbin/httpd (see line 3).
Let’s end the process using kill, remove the execute permissions, and move the files to another location.
$ kill 16324
The kill command ends a process with a given PID, in this case 16324.
$ chmod -x /usr/sbin/httpd
The chmod command lets you edit the permissions to this file. We’ll use it to remove any execution rights it might have “-x”.
$ mkdir /home/tmp-sandbox/ mv /usr/sbin/httpd /home/tmp-sandbox/
We’ll create the directory “/home/tmp-sandbox/” and copy the files there, so we can view them later without risk.
Ok Batman, I stopped the program. Can I go out and play now?
Oh no, not yet.
You may have ended the program, but you still have to prevent if from being started again. First of all, check to see if the program was filed under /etc/init.d . Every command located in this directory will be executed upon boot – and you don’t want the process to restart if you have to reboot the server.
Next, make sure it’s not created a crontab entry to be executed every X-seconds.
$ crontab -l
This will list all your cronjobs, for the current user – probably root. If you see any command in there that is started from the location where you just found your culprit, remove it from the crontab.
$ crontab -e
This lets you edit the cronjobs, just like a normal vi. Remove the line, and save the file (:wq). Also make sure to log in to the user that executed your bad process (su
Next up, update all your software. Either via yum or apt-get, or manually – just make sure it’s all up to date. Also, avoid using tools such as Webmin to manage a server – it’s highly vulnerable to these kinds of attacks.
You might want to check your /etc/, /tmp/ and /home/ folders just in case, for any files that were left there by your hacker. Remember: it’s not because you stopped the process, that you’re safe.
Assume there are other files, still hidden, on your server. Find them, and disable them. Then study their contents, and learn from it.
If this article was in any way useful to you, please leave a message and state how it helped you. If you see errors, of have other tips – don’t hesitate to share!