So, over the last week or so, apache on my mail/web server has crashed a couple of times, leaving itself in a zombie state. This is new behaviour for it, as it’s been stable as for ages. A restart fixed it each time, so meh.
Then this morning, I get an e-mail. Here’s a quote.
Hello from USinternetworking (USi). I am a Security Engineer here trying to track down a security incident that appears to have originated from your network on December 04, 2008. Please investigate a TCP sweep of port 22 from [Redacted] and inform me of the results (account cancelled, user warned, etc). I will require this information in order to close the ticket on this activity. I have attached a portion of the log details as evidence.
Now, this server has exactly one user on it with shell access, and that’s me. And I didn’t do any portscans on these people.
This doesn’t bode well.
So, couple this with apache weirdness, and I start trawling through apache logs, and what do I find? Evidence of a cgi escalation hack, which leads me to irc drop bot files in a crudely hidden directory in the original users home dir.
Dammit. Even though the user directory they were using is chrooted off and they probably didn’t get any further, I have to treat the whole server as compromised and completely reinstall everything. Total pain in the ass.
CURSE YOU, SCRIPT KIDDIES!
December 5, 2008 at 12:17 am
l4m3 D:
December 5, 2008 at 12:36 am
Indeed full of the lame. Having recently being forced to do a complete re-install of my server I feel your pain. Now I’m all paranoid about my machine too. When you have a sec can you send me an e-mail with the location etc of the compromised files so I can check my server out for it too?
Cheers.
Damn script-kiddies.
(goes to turn off all CGI on his machine.)
December 5, 2008 at 1:49 am
The dude wasn’t exactly bunnies about it though, all full of “inform me” and “I require” and not an ounce of “oh shit dude, this sucks but”.
Bums 🙁
December 5, 2008 at 2:13 am
Oh, I presumed it was an automated response. Not likely an actual human is chasing every port scan they get, you’d need a team of thousands.
I replied to it with “The user in question has had their hands amputated, in accordance with New Zealand law regarding e-cyber-criminals. Thank you for your time with regards to this matter.”
December 5, 2008 at 2:15 am
…which strikes me as a STRONGER reason to provide a well-crafted, polite, “human” response.
If I had a robot that could bake cakes, I would never give my friends and visitors bread and butter.
December 5, 2008 at 2:39 am
Dude, you were super-hacked!
December 6, 2008 at 12:42 am
same thing happened to me recently on a production box, and i’m also totally sure we’re kernel rootkitted, which teh sux0r.
December 6, 2008 at 8:31 pm
What was the CGI? Anything that anyone else is likely to be running?
December 6, 2008 at 10:54 pm
VBulletin. An older version, even. So I shouldn’t have really been too shocked that it got pwnd, given that VB is notorious for such things.
I should probably sandbox VB high-threat vectors like that off into virtual machines, and then tripwire the hell out of ’em so that I can tell when they get hacked and just blow them away and re-install from a known good image every time it happens.