Introducing the Oracle-SimpleDB Hybrid

My company would like to migrate its systems to the cloud. As this will take several months, the engineering team needs to support data access in both the cloud and its data center in the interim. Also, the RDBMS system might be maintained until some functionality (e.g. Backup-Restore) is created in SimpleDB.

To this aim, for the past 9 months, I have been building an eventually-consistent, multi-master data store. This system is comprised of an Oracle replica and several SimpleDB replicas. As I near completion of this system, I’d like to share its design.

Here’s the system:

We plan on accepting reads and writes in our data center (Oracle) and in our AWS region (SimpleDB). There are 2 Incremental Replicators (IRs) that transmit the changes between Oracle and SimpleDB. One replicates data from Oracle to SimpleDB, the other replicates data back from SimpleDB to Oracle.

There were many tough challenges in building this system. Not only are we trying to translate data between a key-value store and an RDBMS (2 totally disjoint concepts), we are also trying to keep data in sync in the absence of a global or vector clock. Additionally, pulling large amounts of data out of your work-horse OLTP system is no easy task.

Each of these topics were deep and often led to lengthy debates.

The challenges can be summarized in several parts:

  • Part 1 : Pulling data out of Oracle Efficiently
  • Part 2 : Solving the Oracle-SimpleDB Eventual Consistency Problem
  • Part 3 : Defining the SimpleDB-Oracle translation

These challenges will be discussed in the near future. Feel free to follow me or retweet me (@r39132) on Twitter. Also DM me if you have any questions.

Cheers!

- s -

  1. curs-bnr reblogged this from rooksfury
  2. rooksfury posted this
blog comments powered by Disqus
About Me
A blog describing my work in building websites that hundreds of millions of people visit. I'm a senior member of LinkedIn Search Infrastructure. I previously held technical and leadership roles at Netflix, Etsy, eBay & Siebel Systems. In addition to the nerdy stuff, I've included some stunning photography for your pure enjoyment!
Tumblelogs I follow: