So, over the last week or so, apache on my mail/web server has crashed a couple of times, leaving itself in a zombie state. This is new behaviour for it, as it’s been stable as for ages. A restart fixed it each time, so meh.
Then this morning, I get an e-mail. Here’s a quote.
Hello from USinternetworking (USi). I am a Security Engineer here trying to track down a security incident that appears to have originated from your network on December 04, 2008. Please investigate a TCP sweep of port 22 from [Redacted] and inform me of the results (account cancelled, user warned, etc). I will require this information in order to close the ticket on this activity. I have attached a portion of the log details as evidence.
Now, this server has exactly one user on it with shell access, and that’s me. And I didn’t do any portscans on these people.
This doesn’t bode well.
So, couple this with apache weirdness, and I start trawling through apache logs, and what do I find? Evidence of a cgi escalation hack, which leads me to irc drop bot files in a crudely hidden directory in the original users home dir.
Dammit. Even though the user directory they were using is chrooted off and they probably didn’t get any further, I have to treat the whole server as compromised and completely reinstall everything. Total pain in the ass.
CURSE YOU, SCRIPT KIDDIES!