How well does your MySQL survive power loss?

The huge power outage at 365, affecting such sites as craigslist.org and Yelp, brings to mind some important thoughts about redundancy and infrastructure. Of the many sites at 365, including both new, interesting startups and more mature sites, how many survived the power outage well? More importantly, did they lose power on their databases, and then did they lose any data?

It's easy to believe in your provider when they assure you of uptime, redundant power, excellent cooling, and whatever else they promise to get your business. But you really shouldn't, and this is an example of why. You must have multiple sites! Preferably geographically diverse (nothing hurts like an earthquake or hurricane taking out your main data center and your redundant data center at the same time).

MySQL, sadly, is not a durable database when it comes to power outages. I've heard from more than one person affected by the 365 incident and, consistent with my own experience, MySQL does not handle sudden power outages well. Even with state of the art SCSI controllers with expensive battery-backed write buffering, MySQL can toss it all to the wind, especially when replication is involved. Innodb itself is fairly hardened, but when, for instance, replication state does not match innodb's state (because, sadly, MySQL stores replication information non-transactionally in a separate file), you may have to skip statements. Or maybe you use myisam -- now you have to repair your tables, which takes time. Even innodb can take time to come back from a crash, particularly if you use a large buffer.

So, how well will your database handle a power outage? The 365 outage sucked, but in the grand scheme of things, it could have been a lot worse. If your primary data center lost power for 72+ hours, how well would your business function? Plan ahead! Get those backups and replicas off-site.

iPhone Experience

So, I got myself an iPhone. Pretty predictable to anyone who knows me or reads this rarely updated blog. I had pretty high expectations going into it, and a pretty good idea of what to expect, and I wasn't disappointed. I'll leave it to other bloggers, posters, and websites to detail owning and using the actual device.

I'm more interested in the experience around the launch. I admit, I was lame and had a feeling supplies wouldn't be scarce, so I waited til around 9PM to go to the Palo Alto "flagship" Apple retail store and, sure enough, spent 30 minutes finding parking and 10 minutes in line and purchasing the actual device. So, nothing really exciting there. I did enjoy talking with coworkers who were online whilst inline (not much to do when you're in line at an Apple store, really, except email, IM, and browse the web), and there was quite a bit of email on the topic on enthusiast mailing lists at work, but somehow I just didn't get swept up into the experience itself. I did enjoy watching it, though.

Some people think of it as the coming of a new age of cell phones (admittedly, I do think this will change the industry). Others think of it as a crass demonstration of the worst aspects of capitalism (admittedly, there is some ugliness when you think about just how spoiled this shows some parts of the world are compared to those parts that are much worse off). In the end, though, I think it's just people doing what people do and pursuing the things that matter to them and that give them joy.

One thing that did strike me was Robert Scoble at the Palo Alto store and his impressive 36 hour campout. It paid off -- he got to be first in line, had some nice publicity (even making the cover of the San Jose Mercury News). What really struck me was that he did it with his son, and that they had the opportunity to meet some great people like Andy Hertzfeld, Bill Atkinson, and other Silicon Valley luminaries who also visited to get their own iPhones. Most striking was that he did it with his son, Patrick; sounds like they had a great time, and I'm sure both will look back on it fondly years from now, long after the iPhone is a relic of the first step out of the seemingly prehistoric time of cell phones. Their experience is a nice reminder of some of the human aspect of this spectacle. It isn't just about the iPhone, it's also about the people.

Fun, NewMySQL Storage Engine

In the spirit of the recent MySQL Conference, I decided to have some fun and make a MySQL Storage Engine that would be of practical use. My idea is that so, so much information is available through, say, /proc on Linux and FreeBSD that it is a shame there isn't a convenient way to get to it short of ssh or (shudder) snmp. So, I made a little storage engine that turns arbitrary files into MySQL tables. It supports basic things like space separation to derive columns, etc. Pretty simple, but I'm amazed how easy it was and how useful it could be.

Enjoy!

http://code.google.com/p/mysql-filesystem-engine/

Ideas for files to watch:

• /proc/mdstat - how is your software RAID array doing? any bad disks?


• /proc/megaraid/* - how is your hardware RAID controller?


• /proc/meminfo - watch memory on your MySQL in depth


• /etc/lsb-release - identify your server's OS easily

As an upshot, it gives me a chance to play more with Project Hosting on code.google.com; it's come a long way, and it really made getting code out there much easier than manually setting up SVN, a web site, etc would have.