Today I toyed around with Heartbeat. Not really playing because I was operating on live systems. Recently we have extended our 19″ cabinet with a bunch of new servers, a couple of them are running Xen. But there is one big disadvantage to Xen-hosts, when they freeze you have som major stront-aan-de-knikker (problems). Systems should not freeze, of course not, but in Real Life sometimes they do.
As we have multiple Xen hosts, I setup some new images on different servers and wanted them to take over the service when the other one has died. I had already done that for our caching squid servers earlier who take the load of our main webservers (which is a blessing when digikam.org ends up on some random open source news site). Which worked well, but if the mysql server is down, most sites are dead anyhow.
Today I put up a new image for the ldap connections. Openldap has some nice tricks to replicate to another server and after that I installed heartbeat to float around with the ip-number, so when one dies the other picks up that ip-address.
Then I tried the same for mysql. Things are a bit more complicated, because we are talking about a massive amount of data in our case. For ldap we don’t really care if there is no writing is possible when the master dies (who wants to create new pop-boxes when a server is down?), but for mysql this is not acceptable. Joomla! and other sites won’t show anything if they can not record some user data in the database. So that setup has to work in a way that the slave mysql server will sync untill the master dies and then promotes itself to master. And heartbeat will take care of the ip-number switch. But when the machine is reanimated, the ip-number should not float back to that server, else we would have some major corruptions.
But that’s all done and works. It’s good to see applications have thought about this, provide the possibility to replication and heartbeat being able to do exactly what’s needed to make it high-available.
