Musings of a West Texas Business Venture

Sequoia and C-JDBC

Getting technical for a few minutes.

We are in the middle of a project by which we will be building a clustered web and database solution for a customer. This customer is looking for:

• Failover - Ensuring that if one machine breaks, the next one (or two, or three....) keeps running without any hiccups.
• Load Balancing - If there are two or three back-end servers in the cluster, then the requests to the database should be able to go to any of them.
  
• Reliability - The system can keep running for a long time, and when there is a hiccup (they DO happen), the system can recover fast and keep running.
• Ease Of Backups - Backing up a server can take some time, and as a system grows, so does the backup and restore time. Clusters make this even more difficult.
• Database Independence - If the customer decides that their database is junk, or expensive (like Informix, in this case), they should be able to move to another database with minimal effort.
  

For about a year now, we've been using C-JDBC (now Sequoia) for some time, but hadn't had a chance yet to really kill it in testing. So, we decided we would before we put anything into production using it. Needless to say, it didn't handle the various scenarios that well.

We brought up a few instances of Sequoia, got it reading off of a few databases (in RAID-1 mode), and started hammering it with a few thousand inserts. That worked, but was slow, an average insertion taking about 40% longer than if it was into a single database (we assumed some slow-down on the insertions). Selects, updates, insertions, and deletions all worked as expected.

Things started getting out of hand when we started unplugging machines from the network, and taking backups. It seems that Sequoia uses JGroups in the background to handle the master/slave communication. That's fine. However, we found a lot of problems came up if the network between the master/slave was slow, or dropped packets. The end result was very undefined, and could end up with machines out of sync or un-responsive.

Furthermore, if a machine goes down, and you want to bring it back online, you're in for a headache. The JGroups layer appear to bring the machine back into play just fine, but the machine will need sync'd back up. To do this, you have to use the C-JDBC / Sequoia backuper tool. This backuper uses either Octopus (which we couldn't ever get to work, with or without Sequoia), or the native backup tool (in this case, pg_dump).

So, a Sequoia restore consists of restoring a complete dump, and then restoring the meta-data log that Sequoia keeps. The meta-data log will ensure that the database gets back in sync, without taking down the running instances. This can take hours, or longer. Another issue is specific to pg_dump (something I have an issue with anyway), which cannot take an incremental backup. Instead, it dumps the whole database into a single file (yes, you can split it up, manually) -- and transferring that file back and forth between the nodes (to do a restore) can take some time.

We encountered a lot of instabilities when we took down nodes while it was running. Sometimes we would have to do a full restore. Sometimes we would have to restart the servers entirely, and reload the entire cluster. Overall, the package is fine ... but not yet production ready. A bigger issue becomes support. Community support is OK for a project -- but at the cluster / enterprise level, commercial support is probably a good thing.

We are now looking at native PostgreSQL implementations. However, with the idea of support, we're looking at EnterpriseDB. We already have it up and running, with Slony I, and using pgpool on the front-end as a load-balancer. We're quite happy with it at the moment. It does have its own quirks, however.

More posts later on the PostgreSQL solution.

posted by Anthony Presley @ 9:52 AM