How well does your MySQL survive power loss?

The huge power outage at 365, affecting such sites as craigslist.org and Yelp, brings to mind some important thoughts about redundancy and infrastructure. Of the many sites at 365, including both new, interesting startups and more mature sites, how many survived the power outage well? More importantly, did they lose power on their databases, and then did they lose any data?

It's easy to believe in your provider when they assure you of uptime, redundant power, excellent cooling, and whatever else they promise to get your business. But you really shouldn't, and this is an example of why. You must have multiple sites! Preferably geographically diverse (nothing hurts like an earthquake or hurricane taking out your main data center and your redundant data center at the same time).

MySQL, sadly, is not a durable database when it comes to power outages. I've heard from more than one person affected by the 365 incident and, consistent with my own experience, MySQL does not handle sudden power outages well. Even with state of the art SCSI controllers with expensive battery-backed write buffering, MySQL can toss it all to the wind, especially when replication is involved. Innodb itself is fairly hardened, but when, for instance, replication state does not match innodb's state (because, sadly, MySQL stores replication information non-transactionally in a separate file), you may have to skip statements. Or maybe you use myisam -- now you have to repair your tables, which takes time. Even innodb can take time to come back from a crash, particularly if you use a large buffer.

So, how well will your database handle a power outage? The 365 outage sucked, but in the grand scheme of things, it could have been a lot worse. If your primary data center lost power for 72+ hours, how well would your business function? Plan ahead! Get those backups and replicas off-site.