April 2007 Archives

Over at Solfo I play sysadmin in between being a programmer and DBA. Between real hardware, xen boxes and network equipment we have a few dozen devices. One of the really neat tools we use for monitoring is Munin. It's (yet another) tool to graph system stats and anything else you can write a plugin for.

One of the default plugins is memory use. In this graph you can see the memory usage for one of our application servers. Last week we changed some configuration so the application won't grow to eat a ton of memory every few hours.

Other similar tools are Cacti and Ganglia. I've played with both but they didn't hook me like Munin did. I never really liked how Cacti and Ganglia is put together, but the sensibilities of Munin are ones that makes a great tool.

The basic Munin configuration is extremely short and simple; in particular on the monitored nodes.

Everything is driven by "plugins" - again very simple and self-contained. The plugins "self-configure", for example giving the list of mount names for the disk usage graphs so there's no configuration. That way most of the plugins don't need configuration, making it's easy to just duplicate the configuration to all your nodes.

It's flexible, just in the right spots. And when you do need a configuration override for a plugin it's easy to do. The configuration system even fits well the puppet configuration tool - another favorite.

We run multiple MySQL instances on several of our database servers and the existing MySQL plugins don't handle that, but since all the plugins are shell or Perl scripts, it will be easy to fix.

Yesterday Phillip Reisner was giving an introduction to DRBD. I won't get into it here, but essentially it's a way to keep blockdevices on two different hosts in sync to get a kind of shared disk. (Go read about it on their site!)

I've used it for a while to get high-availability from two NFS servers and I always recommend it in my scalability talk. It's awesome. (Heartbeat which is the "fail-over" software in the typical configuration isn't so great, more on that another time).

Anyway, being at the MySQL conference the context of course is "how can this be used for MySQL". Usually I prefer a simpler master-master replication setup for redundancy. Later in the afternoon Mats Kindahl (one of the replication developers) pointed out how the drbd setup is great for having a highly-available master in a more typical master-slave(*N) setup.

It's funny, just as I write this Paul Tuckfield from YouTube is giving a keynote and telling about the complicated stuff they were doing to deal with failure of their master in their replication system. :-)

If they used drbd the redundant masters would "share the disks" (but you have the content duplicated so there isn't even a disk SPOF). When the primary master fails, the second one would just start up and all the replication slaves wouldn't know the difference. Magic!

My slides from the tutorial are posted on the Develooper talks page. I'll add a link to Jays slides when he posts them.

I am giving a tutorial ("Real-world MySQL Performance Tuning") at the MySQL conference Monday afternoon. I giving it with Jay Pipes and I think it's going to be pretty fun. We are covering quite ways to think about similar problems (and when we have overlap we often disagree! Talk about getting a "two-for-one"! :-) I did a trial run of my longer version for la.pm a month ago and I think I figured out how to pace the length, but we'll see on monday. It's tricky! I had the 45 minute version down pretty well, but in December I gave a one hour version at the Web Builder conference in Las Vegas and that was too little time for the version I had there to be as fun as it should be.

Uh oh, our tutorial is sold out. Cool. (Maybe they gave us the smallest room ;-) )

I am planning to be at the conference Wednesday/Thursday too. If you are there send me a mail or an IM (ask@plys.net on gtalk/jabber, askbjoernhansen on AIM).

A couple weeks ago I got a mail from jobindex.dk (the first Danish job site) that they were going to delete my profile since it had not been updated for 5 years.

They sent it to my email account - still working! - on my very first domain (plys.net).

I logged in and it turned out it wasn't 5 but 10 years ago I had updated my profile there. Woah. The photo is me in 1996.

Spring of 1997 was around the time I started using MySQL, but I didn't mention that (or even mSQL). I had setup my first Linux system (RedHat 3.0.3 with a 1.2.13 kernel) with Apache (1.1!), Sendmail and BIND/named in the summer of 1996 but other than "internet" and "linux" I didn't mention any of that though. In a list of keyword I had:

"Pascal, internet, unix, linux, wordperfect, macintosh, OS/2"

OS/2 Warp! Did anyone else use that? In Denmark it was quite popular for running BBS and Fidonet nodes.

It's curious I didn't have Perl or Rexx on the list either (I was a shark with Rexx and batch programming with 4OS2 back then. Maybe I thought Pascal was the only serious language to add? I wasn't any good at Perl back then ("or now!" the comments come in, ha-ha) so that'd be a good reason not to include that.

What was your first programming language? How did you get bitten by the Linux bug? I am glad I got started on this in "simpler times", but I suppose it will feel that way no matter when you start.

Over at perl.org we've been asked if we could host one of the popular Perl community resources (it's vaguely hardware intensive and the current host is good with the software, but running the hardware for this is a distraction to them). So we try to say yes...

However, we need less hardware - not more, so we've been stalling it until the release of Red Hat Enterprise Linux 5 which has virtualization (via Xen) built-in.

Vani and I went to the datacenter today to replace a bios battery in the mailing list server, a drive in another box and install RHEL5 on a couple of newer generation boxes we have (one of them was one of the awesome HP boxes that AMD gave us some time ago).

We're just getting started with it, but so far so good. We have about 20 boxes (mostly ancient hardware) and not much time to look after them so we really really really like having their 7 years support policy so we don't have to do major upgrades until we want to. To fully explain how awesome this is: Most of the boxes we look after run RHEL, released in 2003. They keep making security and other critical updates available for that until 2010! Our version 4 boxes we don't have to upgrade until 2012.

Sure, we could get those updates with CentOS too (I think), but with RHEL we also get Red Hat Network that makes keeping track of updates incredibly simple.

So, Red Hat: Thank you! :-D