Just before OSCON we got a console server and some power management units from Cyclades (it's awesome, can't recommend it enough!) and a load balancer from Coyote Point. Last week I ordered the cables and adapters we needed and monday afternoon Robert and I went down to the data center to install the new goodies.
Plugging in the power management units required us to shut down everything, or at least it was much easier that way. The upside is that now we can login to all our equipment even if we mess up the firewall rules or kernel configuration on a box.
Of course it ended up taking about a billion hours before we were done setting up power and console cables everywhere and configuring the console server.
A good part of the time went, once again, to dealing with the ancient onion server. It started to spontaneously reboot every 10-30 minutes again. (It takes 20-25 minutes for it to boot with disk check and all, big old raid).
I've just spent a bunch of hours moving things around, fixing them to work on the new systems and setting them up on other boxes so we should be doing okay without onion -- except for one big thing. All the mailing lists and the NNTP service is still living on onion. Not entirely sure what to do if we can't make the box run again a bit more stable. The lists and archives are something like half a million files so it's hard to get copied off in the few minutes I get every few hours. And it's our only FreeBSD box now. Can I mount the UFS file system with a standard RHEL kernel? If so maybe I can hook up the RAID to a linux box and copy it off.
Last time onion did this trick it was only while we had the keyboard plugged in, this time it just won't stabilize.
Anyway, the good news is that we are now able to manage more stuff remotely. The bad news is that the mailing list service will be sporadic for another day or two in worst case.
Zzzzz.zZ... time to sleep for me.
ooh, of course now onion is up and I should try copying some more things off there...
Leave a comment