Practical Cloud Computing

Senior Software Engineer – Cloud Performance - Netflix

Sat, 24 Jul 2010 20:57:00 -0700

Senior Software Engineer – Cloud Performance – Netflix
Los Gatos, CA

The Culture
Netflix hires extraordinary performers and gives them the freedom to make an impact. You may be aware of our booming streaming business, but are you aware of our advanced usage of cloud computing? In anticipation of our business’ rapid growth, Netflix, an early cloud adopter, now ranks among the top users of cloud-based infrastructure-as-a-service (a.k.a. IAAS).

The Position
We are looking for best-of-breed, performance-minded software engineers with a passion and talent for scaling high-traffic distributed systems. You should have experience building similarly-trafficked systems and a track record of improving them. Your improvements should be represented by hard-numbers and grounded in engineering principles.

Responsibilities include
• Drive cloud performance & scalability optimization at Netflix and at our cloud partners
• Proactively define and expose metrics that can improve our services’ performance, scalability, and availability
• As a member of this team, you will also work on parts of our core software
• Define and evangelize best-practices at Netflix for Cloud usage

Minimum Job Qualifications
• 10 years of relevant software engineering experience - 6 years of experience with high-traffic, large-scale distributed systems and client-server architectures
• Experience with Cloud Computing platforms (e.g. Amazon AWS, Microsoft Azure, Google App Engine)
• Object-oriented programming experience with Java
• BS/MS in computer science (or equivalent)

Winning Qualities
• Understands complex systems from a performance perspective
• Works well in teams
• Shows leadership
• Is meticulous and numbers-driven
• Employs unambiguous, crystal-clear communication

Contact me at siddharthanand@yahoo.com with your resume. Principals only (i.e. no recruiters). Locals preferred. Also, implement a method in Java, given input like AAABBaaccC, writes A3B2a2c2C1. Attach the source code with your email. What is the runtime and memory complexity of your solution using Big O notation?

SimpleDB Essentials for High Performance Users : Part 3

Mon, 21 Jun 2010 11:40:00 -0700

This is Part 3 of SimpleDB Essentials for High Performance Users. Check out Part 2

Work around Attribute Value Length Limits
1. If you need to store data that is vastly larger than 1024 bytes in a SimpleDB attribute, consider storing that data in S3 and putting a pointer (i.e. bucket name + object key) to the data in the simpleDB attribute. However, the drawback from this approach is that you will require 2 round-trips (i.e. one to SimpleDB and one to S3) to compose one logical row. Beyond the obvious performance hit, this approach is not transactionally sound.
2. A better approach is to split that data over several SimpleDB attributes. You will need to control the splitting and joining logic of these SimpleDB attributes, but you will only need one roundtrip and you can leverage conditional puts for concurrency control. This approach is ideal if your data can fit in 10 or fewer attributes.
  1. Just remember that subsequent updates to these split attributes might be of different length
Getting tripped up by the Default Select Query pagination limit of 100
1. You must be aware that the SDB Select query supports the “limit N” expression. This allows the developer to specify N up to a max of 2500. If the developer chooses N=200 for example and 1000 items match the WHERE clause conditions, then the results would be returned in chunks of 200 at a time. 5 subsequent round trips would be required to fetch the 1000 items. For customer facing functionality, you are risking end-user timeouts. To avoid this, always specify “limit 2500”. Note: if you don’t specify it, the default value of 100 is assumed by SimpleDB
2. Avoid any client code that auto-follows tokens returned by SimpleDB. SimpleDB Query timeouts could result in an unpredictably long-cycle of next-pointers. Auto-following these can not only result in an infinite loop on your servers, but customer-browser timeouts as well. Instead, follow these next pointers judiciously.
Avoid carrying multi-table relationships into the cloud in the form of multi-domain relationships. Try to denormalize these relationships into single items. Doing joins in the application tier might require multiple round-trips to SDB and open customer-facing functionality to time-outs
Remember that there are no sequences, locks, constraints (except for the uniqueness constraint on the item name), triggers, etc.. in SimpleDB. Don’t expect them

SimpleDB Essentials for High Performance Users : Part 2

Mon, 21 Jun 2010 11:33:00 -0700

This Part 2 of SimpleDB Essentials for High Performance Users. Check out Part 1

Beware of Case-senstivity
1. Since domain names and attribute names are case-sensitive, for all domain and attribute names, use uppercase lettering and separate words with “_”
2. When sharding domains, adopt zero-based index numbering and separate it from the root name with “_”
  1. e.g. MY_DOMAIN_0, MY_DOMAIN_1, … , MY_DOMAIN_99
Shard Domains
1. Since writes are throttled to SimpleDB domains, shard your domains to scale write traffic if you expect to do more than 70 singleton puts/second at any point in the future
2. Also, if you plan to ever need more than 1 billion attributes or 10GB of space, shard your domains — use a rule of 10x for estimating customer-related growth
Avoid Non-Indexed Queries
1. Avoid queries like select * from MY_DOMAIN where SOME_ATTRIBUTE is NULL. These queries are not indexed and can result in an unpredictably long cycle of null results and next-pointer following. If you plan to do this query for customer-facing applications, you must use a pseudo-null. This will allow you to do queries like select * from MY_DOMAIN where SOME_ATTRIBUTE = ‘my-pseudo-null’
Be Aware of Eventual Consistency
1. If you require transactional guarantees for certain data, use Conditional Puts. You can optionally also use Consistent Reads, but it is not required.
Use Batch Puts when possible for optimal write performance
1. If you find that you are doing multiple item puts to the same domain, consider using the Batch Put API instead. You can submit up to 25 items (or 256 attributes or 1MB request size) to a single domain using the Batch Put API
2. The only additional constraint is that the item names must be unique in the request, but this makes sense as there is no implied order of the items in the request
3. The Batch Put is an all-or-nothing API. If it fails, then none of the items have been writtten. If it succeeds, the all of the items have been written. Partial success is not possible.
4. When writing 25 items at a time, I have seen a 20-25x improvement in write-throughput when using Batch Put over Singleton Put

Be sure to check out Part 3

SimpleDB Essentials for High Performance Users : Part 1

Fri, 18 Jun 2010 14:16:00 -0700

Preamble

I’ve been a heavy-user of SimpleDB since January 2009, storing, writing, and reading billions of items. Based on my experience, I’ve compiled a list of best practices and conventions to simplify working with SimpleDB. I’ve divided this into multiple parts to ease readability.

Details

Since sorting is lexicographical, if you plan on sorting by certain attributes, then
1. zero-pad logically-numeric attributes
2. use Joda time (i.e. ISO8601 format & Zulu time zone) to store logical dates as Joda time formats are human readable and lexicographically sortable
When storing dates, it is recommended that you store all dates in Joda time and use a single time zone — I recommend the Zulu time zone (i.e. GMT)
Use a naturally-occurring (sometimes composite) unique key for the item name
1. This can speed up by 2 orders of magnitude (i.e. tens of milliseconds vs. seconds) any queries that would otherwise need to “AND” conditions on several attributes. This is because set comparisons (i.e. those involved in multi-attribute WHERE clauses) are costly in SimpleDB
  1. As a contrived example, instead of select * from MY_DOMAIN where FIRST_NAME=’Sid’ and LAST_NAME=’Anand’ use select * from MY_DOMAIN where itemName = ‘Sid:Anand’
If you don’t have a naturally-occuring unique key, then use a GUID or UUID for the item name. In the RDBMS world, people use DB sequences. In the cloud, UUID or GUID is the way to go. It can be computed local to your SimpleDB client — local is always better.
Favor Composite-Value Attributes to speed up Selects
1. Basically, “AND”ing in the WHERE clause is slow (i.e. see the earlier entry). This is a major bug IMHO and needs to be fixed. For small data sets, the work-around is to create composite-value attribute columns to avoid the costly set intersection

Be sure to check out Part 2

A Java Out-of-Memory Error involving GZIP, Typica, and SimpleDB

Fri, 12 Mar 2010 22:58:00 -0800

UPDATE

I am providing an update here to the root cause.

Overview
I ran into an interesting Out of Memory bug this week. It occurs if you use gzip to send/receive data and under-utilize your Java Heap memory. This land-mine has existed since 2004, though hopefully you will not be bitten by it.

Problem Stack
A Java process was throwing the following Out-of-Memory Error.

JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError".
Exception in thread "SDB WriterPool_4_rentalusers_incremental-thread-1" java.lang.OutOfMemoryError: ZIP004:OutOfMemoryError, MEM_ERROR in inflateInit2
at java.util.zip.Inflater.init(Native Method)
at java.util.zip.Inflater.(Inflater.java:105) 
at java.util.zip.ZipFile.getInflater(ZipFile.java:416) 
at java.util.zip.ZipFile.getInputStream(ZipFile.java:359) 
at java.util.zip.ZipFile.getInputStream(ZipFile.java:324) 
at java.util.jar.JarFile.getInputStream(JarFile.java:467) 
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:165) 
at java.net.URL.openStream(URL.java:1041) 
at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:455) 
at com.xerox.amazonws.common.AWSQueryConnection.(AWSQueryConnection.java:102) 
at com.xerox.amazonws.sdb.Domain.(Domain.java:72)
at com.xerox.amazonws.sdb.SimpleDB.getDomain(SimpleDB.java:202)
....
at java.lang.Thread.run(Thread.java:803)

Figuring it Out

This OutOfMemoryError is not java.lang.OutOfMemoryError: Java heap space. The latter is thrown when the memory occupied by all of your reachable Java objects exceeds the max heap -Xmx setting
Inflater is a JNI class used for gunzipping compressed files. Deflater is its counterpart for gzipping
In the example above, the Typica library is using the Inflater to get to a version file inside the Typica.jar. Every time a batch put or some other call is made and you create a new Domain object, Typica is mindlessly unjarring the Typica jar to get this file. (Why doesn’t Typica cache this version file once?).
There is a limited amount of native heap memory on the system. It must be shared by
1. Your Java program (JVM included)
2. Other user programs
3. Native libraries called by your Java program
The Inflater object takes little memory inside the JVM and quite a lot of memory outside the JVM in the native heap
The Inflater’s finalize() cleans up the memory outside the JVM in the native heap
As you know, finalize() is called after minor and major collections. If a major/minor cycle doesn’t happen for a while, the finalize won’t run and you risk running out of native heap memory

This is what was happening! Since we were GCing every 40-60 minutes, Inflater’s finalizers were not run often enough to clear up the native JNI memory. We ran out of native heap memory.

The bug has been around since 2004. Here it is: http://bugs.sun.com/view_bug.do?bug_id=4797189

Fix/Workaround

If you must use Typica, cache the Domain objects. Every time a Domain object is created, you are unjarring the typica.jar to get the version file. As a work-around, you can also reduce your Heap Memory so that finalizers run more often. As for me, I stopped using Typica.

Other links
Some links that I skimmed and found useful, though they are very detailed:

SimpleDB Performance : 5 Steps to Achieving High Write Throughput

Sat, 02 Jan 2010 19:49:00 -0800

I was recently tasked with fork-lifting ~1 billion rows from Oracle into SimpleDB. I completed this forklift in November 2009 after many attempts. To make this as efficient as possible, I worked closely with Amazon’s SimpleDB folks to troubleshoot performance problems and create new APIs. I’d like to share some recommendations and observations.

Although I have covered these recommendations in depth in a previous post (i.e. link above), I’d like present a more succinct list of recommendations and observations here to maximize knowledge transfer.

Architecture

The architecture consists of a daemon (i.e. IR, for Item Replicator) that reads records out of Oracle and puts them into multiple SimpleDB domains. I’ve actually shown a second IR process that reads data out of SimpleDB for insertion into Oracle, but you should ignore it for the purpose of this discussion. When I refer to IR in this article, I mean the process replicating from Oracle to SimpleDB.

Recommendations

Shard your data
1. You can achieve much higher data access rates to multiple domains than to a single domain. Hence, rather than using a single domain, use multiple. This is because write traffic acts as if throttled or rate-limited at a domain level.
Use slow-ramp up for writing
1. AWS (SimpleDB) doesn’t like bursty writes and will often respond by throttling IR. When your data uploader starts up, have it slowly increase the write rate
Use some sort of back-off strategy
1. I’ve adopted Amazon recommendation for retry intervals (i.e. 250ms, 500ms, 1s, 2s). Essentially, wait 250 milliseconds on first failure before retrying, 500 milliseconds on second failure before retrying, and so on. After the 3rd retry attempt, stick to 2 second idle intervals.
Use BatchPutAttributes instead of the singleton PutAttributes
1. This will get you an order-of-magnitude improvement in throughput
Set replace=false on puts
1. This is the default. If you know that you are strictly always inserting unique records, puts with replace=false will run much faster than replace=true
2. Also, since this is the default, Amazon recommends that users not set replace=false at all

Feel free to follow me on Twitter (@r39132).

"The cloud" Explained for Normal People

Fri, 01 Jan 2010 01:29:00 -0800

If you are like most people in software, you have heard the term “The cloud” but have no idea what it means. If you are industrious enough to buy a book, google the term, or troll twitter for related tweets, you are likely exasperated by the shear marketing buzzword blast you encounter.

To make it easier on you, I am going to tell you what it means to me with a very specific example:

I am helping my company move into the cloud. Specifically, we are going to use most of Amazon’s AWS services.

Definitions

First, some abbreviations and definitions:

AWS
- Amazon Web Services, a division of Amazon focusing on hosting our applications and data
SimpleDB
- AWS’s always-available replacement for RDBMSs. Specifically SimpleDB is their hosted, replicated key-value store that is always available and accessible as a web service
S3
- (a.k.a Simple Storage Service) AWS’s always-available file storage solution accessible as a web service
SQS
- (a.k.a Simple Queue Service) AWS’s always-available queueing service accessible as a web service
ELB
- (a.k.a Elastic Load Balancer) AWS’s always-available load balancing service accessible as a web service
EC2
- (a.k.a Elastic Compute Cloud) AWS’s on-demand server offering accessible as a web service
CloudFront
- AWS’s CDN (a.k.a Content Delivery Network) offering accessible as a web service

All of the services above are pay-as-go (and are very reasonable at that) and are accessible as web services. They are also always-available.

So why does one go about using these services?

Why use these AWS services?

SimpleDB
- Instead of storing your data in Oracle, Postgres, or MySQL, put it in SimpleDB. You will have to give up strong-data consistency and ACID transactionality, but you will gain high availability and throughput.
S3
- Want to store and access files and not worry about losing access to them temporarily or permanently due to hardware or software failures? Store them in S3
SQS
- Want to use a messaging solution a la MQ series or WebLogic’s message queue servers? Worried how much much the hardware and licensing costs will run? Never mind! Use SQS and pay as you go.
EC2 and ELB
- Deciding how much money to earmark for data center maintenance and expansion over the next year? If you are running a start-up, you have no idea how your traffic will grow. Also, if you get a TechCrunch mention, your traffic can sky rocket, only to fall to only slightly-higher-than-earlier levels. Do you spend vast amounts of money on hardware to cover your peak traffic hardware needs? Or do you just live with denying users’ access to your site during peaks? EC2 and ELB will spin up hardware when the traffic peaks and spin it down when traffic subsides. You are charged only for what you use.
CloudFront
- Are CDNs charging an arm and a leg? Do they require accurate estimates of your proposed usage over the next few months? Use Cloudfront and pay as you go.

Basically, by moving these services out of your data center and into the cloud, you:

No longer need to run and maintain a data center. You no longer need the associated staff
Free yourself from some of the tax hassles associated with managing capital expenses (e.g. defining depreciation)
Get a more scalable and available infrastructure without trying to build it yourself

IAAS vs PAAS vs SAAS

Amazon is an example of an IAAS (Infrastructure as a Service) provider. They provide a a means to define and deploy your application and environment (i.e. OS and supporting packages) on demand to any number of hardware instances that you need. They do this via ELB + EC2. They also provide various cloud services (e.g. SQS, S3, SimpleDB, CloudFront, etc…)

Definition of IAAS, PAAS, & SAAS
- http://www.keithpij.com/Home/tabid/36/EntryID/27/Default.aspx
For a full taxonomy of the dizzying array of providers of IAAS, SAAS, PAAS, and Cloud Services, see the following link
- http://peterlaird.blogspot.com/2008/09/visual-map-of-cloud-computingsaaspaas.html

I hope you found this article useful. Feel free to follow me @r39132

HTML 5's Web Sockets explained

Fri, 01 Jan 2010 00:35:00 -0800

I’ve been reading a bit about HTML 5’s WebSockets lately.

First, here are some definitions:

Comet - an umbrella term referring to techniques that provide “server push” using standard browser functionality — i.e. without the aid of specialty browser plug-ins. In practice, Comet in most-often implemented via Ajax with long polling.
Long polling - (from Wikipedia) “With long polling, the client requests information from the server in a similar way to a normal poll. However, if the server does not have any information available for the client, instead of sending an empty response, the server holds the request and waits for some information to be available. Once the information becomes available (or after a suitable timeout), a complete response is sent to the client. ”
Gateway - like a proxy, except gateways don’t alter the requests or response that they ferries between browser and server.

With these definitions out of the way, what are WebSockets?

WebSockets is a new proposal under HTML 5 to provide full-duplex, bi-directional client-server interaction over a single TCP connection. The goal of this proposal is manyfold:

Increasing web server connection scalability - web applications that leverage WebSockets use half the number of web server connections than do Comet-based applications. Comet requires separate upstream and downstream connections.
Simplifing the coding task - the WebSocket API is much simpler to code with than the XMLHttpRequest()

Sounds great! So, how do we use them?

Unfortunately, WebSockets require browser support and currently only Chrome supports them (i.e. since version 4.0.249.0). As a stop gap, a company named Kaazing provides a gateway for your existing browsers.

To learn more about WebSockets, check out the links below:

A good how-to using WebSockets from Tornado’s developer:
- http://bret.appspot.com/entry/web-sockets-in-tornado
A short introduction to Web Sockets from Kaazing:
- http://www.kaazing.org/confluence/display/KAAZING/What+is+an+HTML+5+WebSocket

Website Performance - Why you should care and what you can do!

Wed, 30 Dec 2009 15:57:00 -0800

Why Does Performance Matter?

Oftentimes, people speak interchangeably about web site performance, scalability, and availability. Although these 3 terms are related, they are distinct and unique. Here are their definitions:

availability - what is the total length of time that [some part of] a web site is available during a hour/day/year?
scalability - what is the largest number of concurrent users that a system can handle?
performance - what is the [worst] perceived response time for a single user?

It should be pretty clear that for web sites that charge money (either through ads or via subscription-based services), lesser availability translates into lesser revenue. In the same way that you can lose revenue via an outage, you can lose revenue during traffic peaks if your website cannot handle those peaks.

Hence, just as scalability and availability can reduce your top-line (revenue), can performance have a similar affect?

In 2006 Google’s tests showed that increasing load time by 0.5 seconds resulted in a 20% drop in traffic.
In 2007 Amazon’s tests showed that for every 100ms increase in load time, sales would decrease 1%.
This year (2009) Akamai (a CDN leader) revealed in a study that 2 seconds is the new threshold for eCommerce web page response times.

Hence, it does.

Where Should You Look for Performance Issues?

Ask anyone who has worked on a web site or an enterprise system and they will say “Look at your database!”. Although that is true, the mistake people often make is stopping there. After tuning queries to their satisfaction, engineers seem to ignore 5-7 second website page load times. 80-90% of this time really comes from assembling the web page.

Front-end Engineering Best Practices

About 2 years ago, while analyzing website performance numbers, I noticed that 99% of Tomcat server response times were under 1 second. However, web pages would sometimes take 3 seconds to display. Where was this extra time spent? Luckily, I happened to attend a talk by Steve Souders a few weeks later - he filled in the gaps. Here is a link to his front-end engineering best-practices. It’s a very long read, so use it as a reference:

http://developer.yahoo.com/performance/rules.html

Content Delivery Networks (a.k.a. CDNs)

Most engineers are familiar with the benefits of caching, deploying caches in their data centers to speed up back-end response times (e.g. Memcached, Squid, RamDisk, etc…). However, few engineers recognize that a breed of caches outside of your data center can deliver equivalent performance gains. These are CDNs. Here’s a concise summary of how CDNs help you and a quick how-to on using Amazon’s CDN for the masses, dubbed CloudFront.

http://www.labnol.org/internet/setup-content-delivery-network-with-amazon-s3-cloudfront/5446/

Denial of Service (DoS) : Some Thoughts

Sun, 27 Dec 2009 22:43:00 -0800

About a year ago, I had the opportunity to solve a class of Denial-of-Service attacks that were compromising our availability and scalability. During that investigation, I happened upon a revelation. That revelation led to a solution. I’ve since seen that learning applied to other systems, including Amazon’s SimpleDB, so I wanted to share it here.

Consider the following scenario (also depicted below):

A web client issues an HTTP request to a web site
The web site, upon receiving the request, attempts to determine if the current request is part of a larger DOS attack
1. If so, a defense is executed
2. If not, the web request follows a normal execution of business logic
The web server returns a response to the web client

It is relatively simple and inexpensive for a client to issue an HTTP request. The server on the other hand needs to do a lot more work, which may include:

Reading data from data bases
Running algorithms and computations on that data
Filling caches
Creating an HTTP response

These are more expensive than simply issuing an HTTP request - i.e. what the client does. While a server is fulfilling a client’s request, any of the following could be considered as reserved for that request:

A front-end server connection
A fraction of a back-end (e.g. DB) connection if connections are pooled
A fraction of a middle-tier (e.g. an internal web service) connection if connections are pooled
CPU and memory resources on the front-end, middle-tier, and back-end servers

All of the resources above are in limited supply, so at any point of time, a web site can support a finite number of client requests.

To DoS a site, all a client needs to do is to overrun that limit.

How to DoS a Site?

Find an operation on a website that typically takes a lot of time and bang on it with Curl or wget. For example, search or login might be slow operations as they require a lot of work on the back-end. (Disclaimer: I am not suggesting that anyone DoS any site.)

The principle behind DoS is very simple:

Have the client do much less work than the server. Hence, a laptop in Starbucks can easily issue 4k HTTP requests per second. If those requests are expensive for a website to serve, 4K of those requests may overwhelm it.

DoS Defense

How do you defend against DoS? Follow these rules:

Ensure that your DoS detection logic is run as early on as possible in your server’s request processing stack
Ensure that all resources (except for front-end server connections) are O(1) with respect to request traffic under a DoS attack.
1. This means that under attack conditions, web site resource consumption does not increase — not memory, not CPU, not back-end server connections, nothing. Your inbound client connections a.k.a. your front-end server connections can increase — that’s OK.
If you detect a DoS attack, return the cheapest response possible as soon as possible: e.g. HTTP 503 - Server Unavailable.

The basic principle behind fending of a DoS attack is to make responding to attacking requests as cheap as possible so that the server or server farm can keep up with the client.

My DoS Defense

To solve this class of DoS attacks, I ran a simple algorithm to detect the DoS attack. I made sure that the detection algorithm did not create connections or eat up memory or CPU — in effect, these resources were O(1) or constant. This meant relying on local data.

If I detected a DoS attack, I cached some client information in a local cache. This caused subsequent requests to be locked out — a fast failure response was sent to the client.

SimpleDB Recommended Reading List (12/23/09)

Wed, 23 Dec 2009 22:34:00 -0800

Below is a list of recommended reading to understand SimpleDB and other cloud-related topics. The reading list starts with distributed computing basics and ends with in-depth SimpleDB best-practices.

The Oracle-SimpleDB Hybrid Part 3 : Defining the SimpleDB-Oracle translation

Wed, 23 Dec 2009 21:51:00 -0800

Preamble : See Part 2 : Solving the Eventual-Consistency Problem

When building a SimpleDB-Oracle (i.e. any key_value_store-RDBMS) hybrid system, translating between two very different data models presents a challenge. The challenge expands beyond the obvious ACID vs. BASE differences.

Most RDBMSs support the following features:

Triggers
Stored Procedures
Constraints (e.g. integrity, foreign key, unique, etc…)
Sequences
Sequences used as Primary Keys
Locks
Tables without Primary Keys or Unique Keys or both
Relationships between tables

These do not currently exist in SimpleDB. As we transition our applications to the cloud, the absence of some of these features are more problematic than others.

Unique Constraints and Primary Keys

Sometimes, a relational table might not have a Primary Key, but might have a Unique Key — the Unique Key might be a composite of several columns. In such a model, when translating the RDBMS data to SimpleDB, specify the Item Key to be the RDBMS Unique Key.

If an RDBMS table depends on a sequence, especially if the Unique Key or Primary Key contains a sequence, that can be a real problem in the cloud. A separate distributed sequence generator (e.g. Paxos) will be needed in the cloud as there is no way to generate sequences in SimpleDB. Paxos can potentially be a bottle-neck however.

Another option is to replace the Primary Key sequence (i.e. a number) with a GUID (i.e. a varchar2) in the RDBMS. We can generate GUIDs in the cloud easily as there is no need to gather consensus. This will then make it easy to insert new records in the cloud (i.e. SimpleDB) and data center (i.e. Oracle). It will also make the translation between the two by IR processes trivial.

Relationships between Tables

SimpleDB does not recognize relationships between domains. You have 2 options:

Option 1: Keep separate tables as separate domains and do joins in the cloud application
Option 2: Collapse multiple-table relationships into a single domain

The second option is called denormalizing. SimpleDB, like most key-value stores, does not support join semantics, so one of these 2 options will be needed to overcome this deficit.

The option right for a particular set of data really winds down to replication. If data in the 2 RDBMS tables is altered as part of the same DB transaction, then denormalize (i.e. Option 1). If not, then keep the separate RDBMS tables in separate domains and do an application-level join.

Model All Deletes as Soft-deletes

This has more to do with multi-master replication than the translation between Oracle and SimpleDB. However, just keep it in mind. Hard-deletes can cause some confusion in the following quick succession of a chain of events: create followed by a delete followed by a recreate. A soft-delete is one way to clear up the ambiguity this chain of events can cause when events are received out of order.

Discussion

These were the main pain points I encountered. I could work around many of the others. If you have any questions, feel free to note them here or DM me at @r39132 (Twitter).

The Oracle-SimpleDB Hybrid Part 2 : Solving the Eventual Consistency Problem

Wed, 23 Dec 2009 12:15:00 -0800

Preamble: Read Part 1: Pulling Data out of Oracle Efficiently

Creating the Oracle-SimpleDB Hybrid system is a challenge. For one, it is a multihomed system, accepting writes in both the cloud (i.e. SimpleDB) and in our data center (i.e. Oracle). Secondly, the data center and the AWS reqion (i.e. US-east) are on opposite coasts of the US with network latency ~50-100ms. Thirdly, clocks are not synchronized via network protocols like NTP across WANs. NTP across WANs introduce tens of milliseconds of inaccuracy, which may not be good enough to resolve all forms of conflict.

To build a multi-homed system, we needed to keep our Oracle DB in our data center in sync with our SimpleDB domains in the East Coast region.

This is a tough problem to solve. How consistent do we want the data? What if we shoot for strong-consistency?

In order to build a strongly-consistent link between Oracle and SimpleDB, we could use dual-writes via a 2 Phase Commit (a.k.a 2PC) protocol. However, 2PC over a 50-100ms link would be an availability bottleneck, and hence 2PC is not a viable option. Any consensus protocol would suffer the same short-coming.

Since we cannot achieve Strong Consistency between Oracle and SimpleDB, can we achieve Eventual Consistency?

In an Eventually-consistent system, writes that occur in one place (i.e. SimpleDB or Oracle) are replicated asynchronously by IR processes. This is unlike the synchronous writes that occur in a 2PC system.

The biggest problem then becomes resolving data conflicts. In other words, when parallel writes are occurring to the same row in both Oracle and SimpleDB, how do you settle on a winner?

Our business rules allowed us to adopt the Consistency, Not Accuracy Principle. For most data (i.e. non-financial data and other business critical data), we don’t need to be accurate in terms of a global clock or vector clocks. In effect, we don’t need to ensure that the most recent write wins! Clocks are not synchronized, hence we cannot rely on local times to determine when the most recent change occurred.

All we need to do is to pick the same winner in both Oracle and SimpleDB. We need the 2 data sources to be in sync.

If there is a network partition such that Oracle and SimpleDB cannot communicate with one another, then the 2 data sources will diverge as long as the partition exists. This is unavoidable. However, as soon as the communicate link is reestablished, the system s expected to heal and the two data sources are expected to reach agreement.

One night a few weeks ago, I came up with a solution to this problem in the form of an invariant — a condition that, if maintained, would ensure that SimpleDB and Oracle remained in sync.

Introducing the Consistency Invariant

Consistency Invariant: candidate_row.version > stored_row.version

Any candidate_row being written to either SimpleDB or an RDBMS must have a version number greater than that of the existing stored record. If the candidate row does not have a higher version number, the writer must take exactly one of the following actions:

Abandon the write
Pick a higher version and retry the write

As long as this invariant is maintained in the system, the Oracle and SimpleDB components of this multi-homed system will not diverge. One special condition is imposed on replication : replication must abandon writes if the data it wants to replicate (i.e. candidate_row) has a lower version than the targeted stored_row in the replication destination.

Example:

Start with a record in both Oracle and SimpleDB: ID=1, Some_field=foo, version=01/01/09 12:00:00:000
Imagine a write to Oracle that results in: ID=1, Some_field=bar, version=01/01/09 12:10:00:000
Imagine a write to SimpleDB that results in: ID=1, Some_field=kar, version=01/01/09 12:10:00:001
Oracle to SimpleDB replication is abandoned as the candidate_row.version is smaller than the stored_row.version (i.e. 01/01/09 12:10:00:000 < 01/01/09 12:10:00:001 )
SimpleDB to Oracle replication succeeds so that Oracle now has: ID=1, Some_field=kar, version=01/01/09 12:10:00:001

At the end of step 5, both SimpleDB and Oracle have the same data based on local clock values. Due to clock-skew, the winner may not have been the most recent in terms of an omniscient observer’s eyes.

Implementation

In order to implement this system, we need some functionality in SimpleDB. Amazon’s SimpleDB team is working on making this a reality. Once it comes out, we will be able to build our Eventually Consistent system.

Discussion

Without the anticipated Amazon API, we cannot build an eventually-consistent Hybrid system optimized for AP (i.e. from CAP theorem). We would have had to rely on dual-writes, defeating our goal to be highly-available. This pattern and Invariant will likely be the standard solution in time to come as more companies try to move existing RDBMS-based applications to the cloud.

The Oracle-SimpleDB Hybrid Part 1 : Pulling data out of Oracle Efficiently

Tue, 22 Dec 2009 22:31:00 -0800

Preamble : See my previous post titled “Introducing the Oracle-MySQL Hybrid”

In my previous post, I provided an overview of an Oracle-SimpleDB Hybrid system that I am building. It supports writes to multiple masters, replicates data between masters in single-digit seconds (i.e. in the absence of long-term network partitions), is eventually-consistent, and is designed for optimal AP — it survives Network Partitions and is highly available.

My company already relies on Oracle databases. In order to transition to SimpleDB, we will need to move one application at a time into the cloud while keeping our service running. As this cannot happen overnight, we need to keep both SimpleDB and Oracle in sync.

In Part 1 : Pulling data out of Oracle Efficiently, I’m going to discuss one of 3 methods we have devised to replicate data out of an RDBMS. This method is called Trigger-oriented Incremental Replication (a.k.a TIR) and is depicted below in the bottom gray-box.

Before the Oracle-SimpleDB Hybrid system could go live, we needed to copy a lot of data from Oracle to SimpleDB. There were 2 distinct goals:

Copy historical data from Oracle to SimpleDB - i.e. a one-time data fork-lift
Replicate incremental changes as they occur in the live system

One-time Fork-lift : Phase 1

When you run a production OLTP Oracle table that has millions or billions of rows, a one-time fork-lift could mean capturing a large snapshot of the data and then using SimpleDB’s BatchPutAttributes API to load that large snapshot into SimpleDB. The upload could take several weeks. Also, you might need to involve DBAs and SAs to set the snapshot up on a separate host machine so as to not impact your OLTP machine.

Incremental Replication : Phase 2

Once the fork-lift phase of the data replication is complete, you can start streaming new changes to SImpleDB as they happen in Oracle.

Introducing Trickle-lifting, a novel architecture

Our team used a single architecture to solve both of these goals concurrently. In other words, rather than have 2 distinct phases, our architecture met the fork-lift and incremental replication requirements in a single phase with a single architecture.

Essentially, the gray box on the bottom supported incremental replication (i.e. our second requirement). When we piggy-backed the Trickle-lift Architecture (i.e. the gray box on the top) on top of the TIR architecture, we were able to meet our first requirement as well. The Trickle-lift Architecture is a novel architectural alternative to traditional fork-lifting architectures.

Pros and Cons of Entire Architecture

Pros
- No need for DBAs and SAs to set up a snapshot on a separate host
- Less development work is needed
- Ability to easily tune the data uploading rate
- No need to switch off between separate fork-lift and incremental replication phases
Cons
- Use of trigger for all CRUD operations means that writes to the main table will now take longer and be more expensive in terms of system resources like CPU

Trickle-lifting was used in the billion record upload task (see earlier post)

Introducing the Oracle-SimpleDB Hybrid

Tue, 15 Dec 2009 17:57:00 -0800

My company would like to migrate its systems to the cloud. As this will take several months, the engineering team needs to support data access in both the cloud and its data center in the interim. Also, the RDBMS system might be maintained until some functionality (e.g. Backup-Restore) is created in SimpleDB.

To this aim, for the past 9 months, I have been building an eventually-consistent, multi-master data store. This system is comprised of an Oracle replica and several SimpleDB replicas. As I near completion of this system, I’d like to share its design.

Here’s the system:

We plan on accepting reads and writes in our data center (Oracle) and in our AWS region (SimpleDB). There are 2 Incremental Replicators (IRs) that transmit the changes between Oracle and SimpleDB. One replicates data from Oracle to SimpleDB, the other replicates data back from SimpleDB to Oracle.

There were many tough challenges in building this system. Not only are we trying to translate data between a key-value store and an RDBMS (2 totally disjoint concepts), we are also trying to keep data in sync in the absence of a global or vector clock. Additionally, pulling large amounts of data out of your work-horse OLTP system is no easy task.

Each of these topics were deep and often led to lengthy debates.

The challenges can be summarized in several parts:

Part 1 : Pulling data out of Oracle Efficiently
Part 2 : Solving the Oracle-SimpleDB Eventual Consistency Problem
Part 3 : Defining the SimpleDB-Oracle translation

These challenges will be discussed in the near future. Feel free to follow me or retweet me (@r39132) on Twitter. Also DM me if you have any questions.

Cheers!

- s -

Cloud Tips: How to Efficiently Forklift 1 Billion Rows into SimpleDB

Mon, 14 Dec 2009 21:04:00 -0800

About 9 months ago, I was tasked with fork-lifting a massive amount of data into Amazon’s SimpleDB in a short amount of time. I achieved it. Here’s what you need to know.

If you read-on, I’ll show you how to achieve data upload rates of around 10K items/second

SimpleDB Basics

First of all, if you have 1 billion rows to upload, you will need more than 1 domain. This is because Amazon SDB imposes certain limits on how much data you can store in one domain : see limits

Without digressing too much, figure out your optimal domain sharding scheme for you data growth by keeping the following formula in mind:

Storage Usage = (ItemNamesSizeBytes + AttributeValuesSizeBytes + AttributeNameSizebytes)

This is how Amazon computes your Storage Usage vis-a-vis their 10GB limit.

Note: You might need to ask them to raise your domains per account beyond 100 if you find 100 domains is too few for your data growth.

SimpleDB Throttling

Now that you have created the requisite number of domains (say 100) that you need, you might think that you can just pump all this data into SimpleDB in a few hours,

Wrong!

Amazon imposes a concept of fairness on all users of the system. If you try to execute too many writes, Amazon starts returning 503 - Service Unavailable fast-fail responses to you. SDB throttles you.

How many put’s can you execute per second per domain?

My experience has been: 70 singleton puts/domain/sec

If you had 100 domains and could achieve 70 singleton puts/domain/sec, how long would it take you to forklift 1 Billion records? It would take about 1.6 days.

In reality, the single put rate slows down the more full your domains become, so you don’t get this rate throughout.

Earlier this year, Amazon unveiled Batch Put . This gave me a 20x throughput improvement (I was inserting 20 items per batch put call) on nearly-empty domains. I did not trend how the batch put performed as my domains got more full, but my impression was that it was still much faster than the singleton put.

Now, if you have used PutAttributes or BatchPutAttributes, you will know that you can specify “replace=true” or leave the default “replace=false” option on each item. Leaving the default is much, much more efficient. If you know that you are fork-lifting brand new data, then this is the way to go.

SimpleDB Forklift Experiment

I ran the following experiment:

Data Shape and Domain Count

~100 domains
~10 attributes per item
~10 bytes per attribute
~20 byte key (20 byte item id)
~35 Million records

Other Info

Default Replace=false on all items
Domains were empty to start with
20 items per BatchPutAttribute call
Source data from Oracle instance on west coast, SDB target on east coast

Resulting Performance

All data forklifted in 55 minutes (i.e. Average Rate = 10-11K items/sec)

Conclusion:

You can achieve a pretty good rate of data upload into SimpleDB — 35M records in 55 minutes across the US (west coast to east coast).

For my 1 Billion record task, I ran into a series of issues that kept me from sustaining the 10K items/sec upload rate. These include:

Not being able to use replace=false (I had to use replace=true)
Source Oracle DB was a bottle-neck

If you have any questions feel free to DM me at @r39132 (Twitter)

RDBMS vs SimpleDB Overview

Sun, 13 Dec 2009 21:27:00 -0800

Enter the key-value store, exit the RDBMS

Anyone who has worked directly or indirectly with a relational database will tell you that it would be foolish to build a business that didn’t use one to store your business’s data.

One may argue whether MySQL or Oracle is the better choice, but would someone actually argue that an RDBMS (a.k.a. relation database) was not the best choice for storing your data?

Yes! There is a movement, the NoSQL movement, that is challenging the supremacy of RDBMSs for storing your data!

Some are listed here

Now 10 years after Eric Brewer’s game-changing introduction of the CAP theorem, a mass exodus is starting towards AP (availability + partition tolerance) and away from CP (strong consistency + partition tolerance).

Amazon’s SimpleDB is one such alternative to an RDBMS. Simply put, it is a distributed, replicated, eventually-consistent, always-available, key-value store owned and operated (i.e. hosted) by Amazon’s Web Services division.

In SimpleDB-speak, a domain is a table, attributes are columns, and items are records. So, you can speak about a domain named customer_addresses containing 1000 items (i.e. 1000 customer addresses), such that each item contains attributes like Street Name and Zip.

SDB domains are sparse (i.e. schema-less) and SDB supports the following APIs:

select — a subset of the SQL 92 standard offering order by, but no group by or joins, etc…
getAttributes(key, attributeNameList)
putAttributes(key, attribute_name1=attribute_value1, attribute_name2=attribute_value2, ..)
removeAttributes(key, attributeNameList)

With the exception of the putAttributes() call, omission of the attributeNameList is allowed.

I’ve been transitioning several tables to SimpleDB from Oracle over the past year. Here are some issues I have with it.

Backup-recovery
- They don’t have a way for users to back up their domains. God forbid that you should corrupt your data, you don’t have the ability to rollback to a previous good checkpoint
High-availability
- When a write is committed, you should be able to read your data within seconds. WHEN A WRITE IS COMMITTED. This is the catch. If the node that you are writing to is being hammered (by you or someone else), you will receive ‘503 - Service Unavailable’ responses. In other words, you won’t be able to commit your writes.
- You may have to make several attempts to commit, during which your own site might become less-available.
Inter-region replication
- Currently, SimpleDB is shared between all availability-zones within a region but is not shared across regions. I don’t have a major qualm with this missing functionality, but you might. Essentially, you choose regions in order to achieve fast access to local data. In other words, your east coast customers should access SDB-US-East, while your west coast customers should access SDB-US-West. You could partition your data according to where you customers live (registered) or you could keep your data in sync between East and West regions and route traffic based on IP to the nearest coast. Whatever you choose, you will need to solve this problem.

Benefits of SimpleDB over RDBMS

Even with these deficits, I am quite pleased with SimpleDB. If you are starting a new company and need to decide on an OLTP system. Use SimpleDB. You won’t need to hire System Admins or DBAs. Your engineers won’t need to understand the need for connection pool management, SQL injection, prepared statements, query optimization, or other DB-related performance handicaps.

Your developers will simply make web service calls. There are some best-practices to follow, but they are few and simple to understand.

If you are transition from an RDBMS to a key-value store, your decision is more complex. I’ll write on that at a later time.

Eventual Consistency Explained for Techies

Sun, 13 Dec 2009 19:12:00 -0800

Preamble: Have a look at my previous article titled “Eventually Consistency Explained for non-Techies”

Eventual and Weak Consistency

SimpleDB is eventually consistent. Eventual consistency is a version of weak consistency — you may not see the latest writes committed to the system.

Imagine that you have a system of N nodes. Of these, W nodes are involved in any write sent to the system and R nodes are contacted on any read from the system. Strong consistency can be achieved if R+W > N. In other words, if the read sets and write sets overlap, the read can discover the most recent write to the system.

Hence, in order to achieve consistent reads, there are 2 sides to this equation. You can either increase W or increase R to achieve the overlap. There are hence 2 extreme cases:

Fastest Reads: R=1, W=N
- To read the latest write, only 1 node is contacted. However, writes need to be confirmed at all nodes before they are ACKed back to the client. Write performance suffers.
Fastest Writes: R=N, W=1
- Only one node is ever involved in a write. All nodes are contacted for the read and a quorum needs to be reached among all nodes. Read performance suffers.

In an eventually consistent system, R+W < N. Read sets and Write sets are not guaranteed to overlap. Reads will not see the latest write in these cases. However, a gossip-style, lazy data propagation mechanism replicates writes to all nodes. Eventually all nodes will be consistent. In cases where there are conflicting versions for a datum, the system will choose one.

Eventual Consistency Explained for Non-techies

Sun, 13 Dec 2009 18:46:00 -0800

If you work in the Computer industry, especially the Internet industry, chances are good that you have encountered an eventually-consistent system.

For example, when managing an internet or IT business, you might have considered one of all of the following DB architectures:

Use a single DB host
1. e.g. MyHost
Use a single DB host for your writes, but several for your reads
1. e.g. MyWriteHost & MyReadHost1, MyReadHost2, MyReadHost 3, etc …
Use multiple DB hosts
1. e.g. MyHost1, MyHost2, etc….
Use multiple DB hosts for your writes and your reads
1. e.g. MyWriteHost1, MyWriteHost2, etc… & MyReadHost1, MyReadHost2, etc, ….

These 4 choices represent an increasing degree of data traffic partitioning, with 1 having no partitioning and 2 having the most partitioning.

In most eCommerce sites, the DB read traffic dwarfs the DB write traffic, typically 10-to-1. In such cases, architecture 2 makes a lot of sense. For example, a user might change his phone number from 555-5555 to 666-6666. 666-6666 will be written to MyWriteHost. An asynchronous back-ground task will copy data from MyWriteHost to MyReadHost1 (for example) after a brief delay, also known as the Inconsistency Window. If the website tries to read the latest phone number from MyReadHost1, it will still see 555-5555 until the Inconsistency (time) Window expires. This is an eventually-consistent system.

This is known as a read-write split architecture - companies like eBay heavily leverage such an architecture. Read-write split architectures are eventually-consistent.

Another use-case that exhibits eventual-consistency is seen in Amazon’s SimpleDB. Such a system is actually an Active-Active HA storage system. Each node can fail-over to any other node and nodes are kept in sync by an asynchronous background thread.

The "Consistency, Not Accuracy" Principle

Fri, 11 Dec 2009 16:56:00 -0800

Preamble: Read my post “The CAP Theorem distilled”

In my previous post, I started talking about the “Consistency, Not Accuracy” Principle (a.k.a. The CNA Principle)

Essentially, in order to scale your web site and to keep running amidst unpredictable network and system outages, you need to have a replicated, fault-tolerant data store that accepts reads and writes in multiple locations. One replica might be in California and another might be in Virginia. If California were to fall into the great Pacific, your web site should still work and your users should be none the wiser.

However, this distributed data store design now presents the following predicament: How do you keep your replicas in Virginia and California in sync? Also, what does “in sync” actually mean?

As described in my earlier post, throughput and latency don’t scale for 2-phase commit (a.k.a. 2PC) or other forms of pessimistic synchronous replication (e.g. fault tolerant causal broadcasts). In order to have a highly-available system, some portion of writes need to be executed asynchronously. Hence some number of data inconsistencies will arise after writes commit.

Resolving these inconsistencies, a complex task, can be made simpler if we accept the CNA principle. Essentially, it is more important for the data to be consistent between these two data store replicas (i.e. Virginia and California) than for the data to reflect the most recent update to the system (i.e. be accurate).

For example, say you changed a blog entry at noon and then again at 12:01 pm. Assume that the modifications were different. According to this principle, it is possible for one of your changes to be lost. Also, the one that is lost might be the most recent edit.

A system in which accuracy is required, such as a financial application, is vastly more complex and can require causal ordering (e.g. vector clocks), read repair, quorums, etc….

Building a system that follows CNA is vastly simpler to understand and implement. If your site’s business model supports it (which it probably will), then I recommend it.

In a future post, I’ll go into my specific experience with the CNA Principle.