Incident Recovery

An intrusion is not the end of the world. Recovering from an intrusion can be a chore, but is not impossible, if you know what to do. This article will look over intrusion recovery and take a brief look at computer forensics -- i.e., what to do if you want to try to get the law involved in the incident.

Much like any other part of intrusion response, recovery from attack starts before you've been attacked. It can be very difficult to recover if you don't have recent backups of your system -- back things up regularly; nightly if possible. If you've got important information on your system, a nightly backup just makes sense. I prefer backing up to tape if you can afford a tape drive, but it's not a requirement. What you do need is some form of backup that holds your important system files and binaries, so you can restore if something happens, or a rescue disk that contains clean versions of important system binaries. Also (preferably at the time when you actually install your operating system, so you're sure it's clean), run a program that checks your system. Tripwire, for instance, is a wonderful tool that works on Solaris, Linux, and Windows NT. It takes a sort of snapshot of your system and creates a database which contains the checksum, creation date, and access permissions for each file. If you feel that your system may have been compromised, you can run tripwire against it again, and compare the results to see if anything's changed (tripwire can even be run regularly, to detect changes; perhaps run it just before your regular backup, to see if anything's been altered since your last backup). Tripwire aids in recovery because it can point out exactly which files were damaged or altered, as in the case of trojaned binaries and rootkits, and can allow quick replacement of them with good copies. And of course, the third thing to do before an intrusion begins is to be aware of intrusion detection tools. Run them, watch the logs, and be alert -- you can't recover from an attack you don't know happened.

As I discussed in the preceeding article, one of the most important things you can do is log, and maintain the integrity of your logs. The need for good logs really comes into play here, in several different ways. For the purposes of this article, I'm assuming that you have intrusion detection tools running, tripwire installed, and are watching and recording your logs. I should note that it's best to log to a remote, secure loghost, log to a printer, or at least make sure that if your logs are on the same host, your log files are append-only (only new text can be added) -- most rootkits now go through and edit logs to remove an attacker's traces. If you're logging to a different machine or a cheap dot matrix printer in the corner, they'll have a hard time covering their tracks -- I'm going to assume, for this article, that your logs are intact.

When your intrusion detection software starts sending out alerts, the most important rule is -not- to panic. You cannot react faster than data can come in to your computer -- by the time you've noticed the attack, the attacker is already several steps ahead of you, and may already be in your system before you can react.

Isolate the machine. There is one school of thought that advocates pulling the power cord out of the computer (don't shut down first; many rootkits install cleanup routines in the system's shutdown proceedures, and you'll lose anything they'd added). I don't advocate this as a first step -- I suggest pulling the network cable (modem, ethernet, whatever you have connecting your machine to the internet). Pulling the power can lose you a lot of information that would be helpful in diagnostics -- a lot of scripts put files in /tmp, for instance, and on some operating systems that would be lost on a power-down. Once you've gotten the information you need, run tripwire to get a new database of exactly what the system looks like now...-then- pull the power cord (again, don't shutdown normally). The reason for this becomes important later.

It's at this point that you need to decide whether or not to pursue legal action. In most cases, especially for home users, the hassle of law enforcement involvement is not worth it, and all you'll want to do is rebuild and secure your machine. At the end of this article is a brief discussion of what to do if you -do- want to involve law enforcement.

You've been attacked -- now it's time to rebuild. You have two options -- the easy way and the paranoid way. The paranoid way is pretty self-explanatory: wipe -everything- and restore from a known clean backup or reinstall from read-only media. For the easy way, turn your system back on, but -don't- plug it into the network. Get your clean backup disk (run tripwire on your backup to make certain it's clean), find the files that were altered (compare the backup's tripwire database with the current files on your system), and replace them with the safe binaries you'd had on your backup. Commonly replaced binaries include /bin/login, /bin/ps, /bin/ls, /bin/df, /usr/sbin/in.telnetd, /usr/sbin/inetd, /usr/sbin/in.ftpd, /usr/etc/in.tftpd, /usr/sbin/ifconfig, etc (note that these locations may be different for different flavors of UNIX; files may be in /usr/etc, for instance). Check everything -- files can be changed in unexpected ways, or be added in unusual places. Some attackers like to hide their files, for instance, in /tmp, /etc/tmp, /var/tmp, /usr/spool, etc. Look for files with spaces in the name. Look for alterations of /etc/hosts.equiv, /bin/.rhosts (or any .rhosts file at all), /etc/passwd, /etc/group, etc. 'Find' is a good command for this; it can be used to find all suid/sgid files, sneaky .rhosts, etc. Look for suid root binaries in unexpected places. Next, look to make sure there isn't a sniffer installed. On UNIX-based machines, if a sniffer's installed on an interface, the interface will have the PROMISC flag set (short for 'promiscuous' -- it means the interface is listening to all the traffic on the network, not just the packets intended for that interface). Sounds easy? Not so. There are scripts that install a sniffer -and- hide the PROMISC flag. The way to check is to set the interface PROMISC yourself, and then check to see if the PROMISC flag shows up. If it doesn't show up, you may have a problem -- make sure you replace ifconfig with a known good copy, and again, look for strange suid binaries and files owned by root that shouldn't be.

Personally, I recommend wiping everything and starting from a good backup. It's safer, and you don't need to worry about having missed something important. Reinstalling from known good media may be paranoid, but it obviates a lot of the work of finding all files that have been tampered with, and will remove things like sniffers and back doors. If you'd like to do an in-depth analysis, make a complete sector-by-sector copy of the compromised disk before you wipe it, then mount (don't boot, mount) the copy read-only on a known good system, and do your analysis there.

Now start going through the logs. What happened? Do you see anything unusual? Look as far back through your logs as you can; maybe you'll see the initial intrusion. Using another computer (or using the hacked system, if you feel lucky, punk), use search engines to look up anything in your logs you don't understand; you may be surprised at what you find. The goal is to find out exactly how the attacker got in, find the hole, and repair it. Most commonly-exploited holes have patches -- do some research on your favorite search engine, find the exploit, find the patch, and fix the hole. If you can't find anything that might have been used against you (and be sure to look in the Bugtraq archives -- click on 'search' and then select 'bugtraq'), you may want to send email to notify the CERT team. They may not respond, but if it really is a new exploit, they'll look into it -- see Cert's page for more information. Once you've got your system patched and replaced all the altered files, change all passwords on the system, just in case the attacker has your /etc/passwd (or /etc/shadow) file.

But suppose you -do- want to take the matter to law enforcement. The most important thing you can do, if that's the case, is to preserve evidence...and your hard drive is evidence. When you pulled the power on your hacked machine, you preserved as much of the current state of your system as you could. Now you need to physically remove the hard drive from the computer, set it to read-only (if you want to do analysis on it, make a complete physical sector-by-sector copy, and mount it -- don't boot it -- read-only on a known 'good' system, and do your analysis on -that-), and place it in a safe along with a copy of the original tripwire database and a copy of the tripwire database you'd taken just before pulling the plug. Also put into the safe all of the relevant logs, also in read-only format to prove they have not been edited -- one idea is to print out your logs, sign and date them, and have them notarized to prove the electronic copies have not been tampered with. Include as well as much information as you had been able to gather about the attacker (see the previous article for a simple discussion of how to identify the attacker). The more information you can provide law enforcement, the more likely they are to be able to prosecute. Document everything you possibly can -- a clear chain of evidence must be compiled before you can hope to have anything done, and you must be able to show that that evidence has not been tampered with. Once you have all your evidence, contact law enforcement -- I should note that, just as it's a good idea to know who your ISP's security team is ahead of time, it's handy to know ahead of time who to contact among local law.

And be nice to them. You -want- them to like you.

Comments to this article are welcome -- not everyone responds to incidents in the same way, and I'd be very interested in hearing other methods, or hearing opinions I may not have considered yet.

(thanks to dpr@nwo.net for his help with this article)



/dev/null