Flashes of Panic

It’s been there nearly a year, long enough that I don’t even see it anymore, just like the “HOWTO Run a Microsoft-free Shop” printout and the email from someone who thought our website was the best she’d seen for reviewing and browsing through textbooks. (She must not get out much.) But lately people keep noticing it and commenting as though it’s new. It’s a bright-orange sticky note, about two inches square, containing only three lines to make two down-cast eyes and a frowning mouth.

It’s the sad-server sticky. I got home from the Boston Marathon to find chkrootkit putting up yellow caution flags, and a quick day’s forensics with e-mail help from another user of the same server model proved that a lot of system binaries weren’t what they were supposed to be.

It was a pretty clumsy kit, and it shouldn’t have been so easy to find. I started out thinking I could zip it up, replace the compromised binaries, and move on. Then I realized that I couldn’t reliably tell which ones were the compromised binaries, because all the tools which tell me about them… could be compromised. I squawked briefly, then yanked the server offline.

This was the start of a sleepless week for me, since the server in question was our internet gateway, our mail server, and our file sharing server, and I needed to at least fake two out of three until the original box was back on line.

The box went to an outside contractor in Connecticut. On its way out, it was sitting briefly on the reception desk; before I picked it back up for its drive to Connecticut, someone attached the “sad server” sticky.

The contractor fixed the problem by putting a new (second) hard disk in the server, five times larger than the original, and installing the operating system from scratch on the new HDD, mounting the old one only to copy old files. He also juiced up the memory and installed a bunch of nice security apps; learning how to use those was how I learned how to secure the web server we put on line four or five months later.

Meanwhile, I ripped apart retired desktop systems in the office to find enough parts to build Frankenserver. A recently retired Pentium II got an additional network card (one for the internet, one for the LAN) and a fresh Linux installation. Then I had to configure it to be our gateway router. Then I had to set up mailboxes and teach it how to accept mail for our domain. Then I had to teach all our users how to get mail from the temporary box… and then do it all over again when the restored server came back on line. I worked through the weekend and late into the evenings, a true rarity considering what a M-F 9-5 office this is. If it hadn’t been for my other, other job and the fact that the internet connection there was stable, I probably wouldn’t have gone home at all for a few days.

I still don’t know how they got in. It’s enough that they’re out now. That’s why I now enforce tough passwords, why I firewall like mad and prefer to keep our FTP server on an entirely separate box (among other steps.) Even though I learned a lot in a hurry, and came out smarter and more confident, I’m in no hurry to deal with another sad-server sticky.