Practical Cloud Computing

The State of NoSQL in 2012

Thu, 19 Jan 2012 00:47:00 -0800

Preamble Ramble

If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”.

In 2007, Amazon published a paper on Dynamo. The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud.

It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing.

Huh?

A month into the investigation, we start wondering about our Oracle database. How are we are going to move it into the cloud? That’s when we are told that we are to abandon our RDBMS too. Instead, we are going to focus on “AP”-optimized systems. “AP” or high-availability systems is our focus in 2008-2010 as Netflix launches its video streaming business. What’s more available than TV right? Hence, no downtime is allowed, no excuses!

Fast forward to the end of 2011: the past 3 years have been an amazing ride. I helped Netflix with two migrations in 2010: the first was from Netflix’s datacenter to AWS’ cloud, the second from Oracle to SimpleDB. 2011 was no less exciting: with a move from SimpleDB to Cassandra, Netflix was able to expand overseas to the UK and Ireland. Since Cassandra offers configurable routing, a cluster can be spread across multiple continents.

Fast forward to today: I’ve completed my first month at LinkedIn. I’ve spent this month getting familiar with the various NoSQL systems that LinkedIn has built in-house. These include Voldemort (another Dynamo-based system), Krati (a single-machine data store), Espresso (a new system being actively developed), etc… LinkedIn is now facing similar challenges to the Netflix of 3 years ago. Traffic is growing and both Oracle and the datacenter face potential obsolescence.

NoSQL and Cloud Computing to the rescue?

The State of NoSQL Today

Ignoring the datacenter vs. public cloud question for the time-being, what would I pick today regarding a NoSQL alternative to Oracle? Not many people get a chance to solve the same tough problem twice.

For one, there are still quite a few NoSQL alternatives in the market. Some are supported by startups (e.g. Cassandra, Riak, MongoDB, etc..), some are supported indirectly by companies (e.g. LinkedIn’s support of Voldemort, Facebook & StumbleUpon’s support of HBase), and some are supported directly by companies (e.g. AWS’s S3, SimpleDB, and DynamoDB, etc… ).

In making a decision, I’ll consider the following learnings:

Any system that you pick will require 24-7 operational support. If it is not hosted (e.g. by AWS), be prepared to hire a fleet of ops folks to support it yourself. If you don’t have the manpower, I recommend AWS’ DynamoDB
Just because the company got by with one big machine for Oracle, don’t be surprised if the equivalent NoSQL option results in 36 smaller machines. All complete solutions to fault-tolerance support “rebalancing”. Rebalancing speed is determined by data size of a shard. Hence, it’s better to keep the size per shard reasonable to minimize MTTR in times of disaster.
Understand the limitations of your choice:
- MongoDB, at the time of this writing, has a global write-lock. This means that only one write can proceed at a time in a node. If you require high write-throughput, consider something else
- Cassandra (similar to other Dynamo systems) offers great primary key-based access operations (e.g. get, put, delete), but doesn’t scale well for secondary-index lookups
- Cassandra, like some other systems, has a lot of tunables and a lot of internal processes. You are better off turning off some features (e.g. Anti-entropy repair, row cache, etc…) in production to safeguard consistent performance.

Many of the NoSQL vendors view the “battle of NoSQL” to be akin to the RDBMS battle of the 80s, a winner-take-all battle. In the NoSQL world, it is by no means a winner-take-all battle. Distributed Systems are about compromises.

A distributed system picks specific design elements in order to perform well at certain operations. These design elements comprise the DNA of the system. As a result, the system will perform poorly at other operations. In the rush to win mindshare, some NoSQL vendors have added features that don’t make sense for the DNA of the system.

I’ll cite an example here. Systems that shard data based on a primary key will do well when routed by that key. When routed by a secondary key, the system will need to “spray” a query across all shards. If one of the shards is experiencing high latency, the system will return either no results or incomplete (i.e. inconsistent) results. For this reason, it would make sense to store the secondary index on an unsharded (but replicated) system. This concept has been utilized internally at Netflix to support internal use-cases. Secondary indexes are stored in Lucene to point to data in Cassandra.

LinkedIn is following the same pattern in the design of its new system, Espresso. Secondary indexes will be served by Lucene. The secondary index will return rowids in the primary store.

This brings me to another observation. In reviewing Voldemort code recently, I was impressed by the clarity and quality of the code. In core systems, whether distributed or not, code quality and clarity goes a long way. Although Voldemort is an open source project and has been for years, a high degree of discipline has been maintained. I was also impressed by the simplicity of its contract - get(key), put(key,value), and delete(value). The implementors understood the DNA of the system and did not add functionality to the system that is ill-suited to the DNA.

In a similar vein, Krati, Kafka, Zookeeper, and a few other notable open-source projects stick to clear design principles and simple contracts. As such, they become reusable infrastructure pieces that can be used to build an Distributed System that you need. Hence, the system we end up building might be composed of several specialty component systems that can be independently tuned, in some ways similar to HBase. As a counter example, to achieve predictable performance in Cassandra without a significant investment in tuning, it may be easier to turn off features. This is because multiple features in a single machine contend for resources — since each feature has a different DNA (or resource consumption profile), performance diagnosis and tuning can be a pain point.

Till next time, stay tuned!

Netflix & LinkedIn

Sat, 10 Dec 2011 16:52:00 -0800

After 4+ years at Netflix, I am about to embark on a new challenge. On Monday, I start at LinkedIn as a senior member of the Service Infrastructure team. It has been my privilege and honor to work at Netflix these many years. To date, Netflix is the best company that I have worked at.

Netflix

Central to the company’s success is its unique culture of “Freedom and Responsibility”. This culture places decision making, risk taking, innovation, and operational judgement in the hands of each engineer. Management serves to prioritize the big tasks and sets the context for each engineer, leaving each engineer to prioritize his or her own work. There is an allergic aversion to process and micromanagement.

Progress is made by sheer drive and motivation. Accountability is achieved by having each engineer accountable for his or her own work. Gaps are filled not by instituting process but by leveraging automation (e.g. our Simian Army).

This unique culture attracts and rewards folks who are “do-ers not say-ers” and fully-formed adults who play nice with others. In other words, “brilliant jerks” tend to have a short shelf life at Netflix, whereas someone with a good blend of leadership, technical savvy, and social intelligence will go far.

LinkedIn

Frankly, it is this culture and my relationships with cherished colleagues that have kept me from considering positions at other companies for so long.

Fast forward to my acceptance of a position at LinkedIn. If Netflix were to have a peer in terms of technical vision, engineering drive, and a bright future, it would be LinkedIn. We all know LinkedIn as the place to find jobs or employees, manage professional contacts, and follow technology trends, but I believe that feature set to be the tip of the iceberg. Speaking to the technical attractiveness of the company, LinkedIn has long been a leader in big data and analytics, open sourcing technology with some regularity. With renewed post-IPO vigor and an infusion of new talent, they’ve bolstered their commitment to open source and technology evangelism.

As a new entry to their Service Infrastructure team, I hope to share my experience regarding the following:

the development and evangelism of centralized libraries and services
the transition to high availability databases
best practices around high availability and high scalability

A large part of my role at Netflix involved the migration to a public cloud, but it is not clear whether this is something that LinkedIn finds useful.

I look forward to working with Kevin Scott, Mohak Shroff, and their talented teams.

Slides from my QCon SF 2011 talk on Fault Tolerant Systems at Netflix

Sun, 13 Nov 2011 18:49:00 -0800

Abstract : http://qconsf.com/sf2011/speaker/Siddharth+Anand

Keeping Movies Running Amid Thunderstorms!

View more presentations from Sid Anand

"Head in the Cloud" Article about Netflix in the Cloud

Thu, 03 Nov 2011 18:59:00 -0700

Check out the latest Cornell Engineering Magazine Issue for an article on how Netflix leverages the cloud! The story starts on page 10.

Full Version of “The a la menthe” Translation of...

Fri, 09 Sep 2011 00:09:36 -0700

Full Version of “The a la menthe”

Translation of Lyrics : http://www.williamsonday.com/morocco/archives/2007/02/boy-i-remember.html

"NoSQL @ Netflix, Part 2" Talk at OSCON 2011

Wed, 27 Jul 2011 10:01:52 -0700

I’ve been having a great week so far at OSCON. Excellent talks from Riak, Facebook, 10gen, DataStax, Acunu, StumbleUpon, etc….

Uploading slides from my talk here..

OSCON Data 2011 — NoSQL @ Netflix, Part 2 View more presentations from Sid Anand

NoSQL @ Netfix @ OSCON Data, July 25

Sun, 10 Jul 2011 18:31:34 -0700

During July 25-27, O’Reilly will be dedicating a special section of their OSCON open source conference to big data use-cases and technologies. Tracks cover a range of topics, from NoSQL to relational technologies. Additionally tracks dedicated to Hadoop and Visualization are also included. This section is aptly named OSCON Data.

I’ll be speaking during the Monday, 10:40 am slot about Netflix Data Architecture in the cloud. Specifically, I’ll cover how we moved from Oracle to a range of cloud based data storage, including SimpleDB, S3, & Cassandra. Our experience with Cassandra is still in the formative stage, so I will cover our learnings and future plans for the technology.

Be sure to check out my talk

Sid Anand, Cloud Data Architect, Netflix

Epidemics and Gummy Bears

Wed, 06 Jul 2011 14:13:51 -0700

Epidemics and Gummy Bears

Acunu - Cassandra optimized (SSDs, new data structure)

Mon, 27 Jun 2011 22:14:00 -0700

]

For some workloads, they achieve 2 orders of magnitude better latency results relative to a vanilla Cassandra distribution. This is achieved through a combination of SSDs and a new data structure (Stratified B-tree).

Acunu White paper. http://bit.ly/jQc423

Short Blog entries on Cassandra performance

http://bit.ly/mUqWLO Cassandra under heavy write load, Part 1

http://bit.ly/mMT0Ts Cassandra under heavy write load, Part 2

Remembering B-trees http://bit.ly/lH8fwh

http://bit.ly/lryYbf Copy-on-Write B-tree beaten by Stratified B-tree

http://bit.ly/iJg01H A summary of the above paper

http://bit.ly/jV5j1x More on why Fractal-tree DBs scale better on SSDs. Cache oblivious Fractal Trees

http://slidesha.re/jIRkYj More on streaming data structures : doubling array, fractional cascading, and stratified doubling arrays explained (presentation) <—- V. Good

Recent GigaOM Structure 2011 Cloud Guru Panel Video (6/22/2011)

Thu, 23 Jun 2011 13:20:35 -0700

Recent GigaOM Structure 2011 Cloud Guru Panel Video (6/22/2011)

Netflix @ DataStax SF 2011 - Monday, July 11

Mon, 13 Jun 2011 11:26:00 -0700

Netflix will be sponsoring DataStax SF 2011 on Monday, July 11. Though Netflix does not often sponsor events or recruit at conferences, we hope to engage with others active in the Cassandra and Distributed Systems community.

Stop by our booth to learn about what we are working on! Bring your resume as well!

Sid, Cloud Systems, Netflix

Some upcoming conferences that I will be speaking at ...

Fri, 03 Jun 2011 01:25:00 -0700

Netflix for Mobile in the Cloud

Tue, 31 May 2011 22:08:00 -0700

Last week, I attended Intuit’s annual CTOF (Create The OFfering) conference in San Diego. The focus this year was on mobile best practices. I was happy to represent Netflix at this venue and learn from other companies with a foot in either mobile, the cloud, or both.

Netflix is an example of both with support on several mobile platforms (e.g. iOS, Android, Windows Mobile) and traffic served from Amazon’s Cloud.

Intuit CTOF 2011 - Netflix for Mobile in the Cloud View more presentations from Sid Anand

If you are interested in positions at Netflix, working on either the Cloud or mobile, we are hiring!

If you are just stopping by for your Calvin and Hobbes fix, enjoy!

Distributed Systems & Dev Ops Engineers, Netflix Wants You!

Mon, 23 May 2011 16:57:00 -0700

Hi Folks!

As many of you know, Netflix has been expanding its video streaming offering throughout the US and the world. We have been successful thanks to our move to Amazon Web Services’ cloud and our adoption of NoSQL solutions.

-s

Talks, Videos, and White Papers

Wed, 23 Mar 2011 20:51:00 -0700

I thought that I would put up a list of previous and upcoming talks that I will be giving. These talks will tend to focus on my work in Distributed “Cloud” Computing and NoSQL.

Recent & Upcoming Speaking Appearances

QCon SF 2010
Cornell Silicon Valley 2010 Cloud Panel
Silicon Valley Cloud Computing Group (Meetup) — “NoSQL @ Netflix”
QCon London 2011 (2 talks)
Cornell Silicon Valley 2011 Cloud Panel
Intuit CTOF 2011 (private)
GIGAOM Structure 2011 (Panel)
OSCON 2011
eBay Architecture Forum 2011 (private)
QCon SF 2011
Silicon Valley Cloud Computing Group (Meetup) — “Keeping Movie Running Amid Thunderstorms”
Ooyala 2011 (private)
Cornell Silicon Valley 2012 (Johnson School Event)
QCon London 2012

White papers

Netflix’s Transition to High-Availability Storage Systems

Magazine Articles

Patents

System and Method for Building a Point-In-Time Snapshot of an Eventually-Consistent Data Store (WIP)
Data Synchronization between a Data Center Environment and a Cloud Computing Environment (WIP)

Videos

NoSQL @ Netflix talk at Facebook (Feb 2011)

Structure 2011 Guru Panel (June 2011)

NoSQL @ Netflix : Part 1

Sun, 20 Feb 2011 17:07:13 -0800

Hi Folks!

I had a great time this past Thursday (Feb 17, 2011) speaking at the Silicon Valley Cloud Computing Group Meetup at Facebook. The talk covered Netflix’s move from RDBMS to NoSQL, specifically SimpleDB. Subsequent parts will provide our experiences with Cassandra, HBase, and other technologies.

Video is now available. The first 10 minutes are from sponsors, VMWare, RackSpace, and Scalr!

-s

Silicon Valley Meetup & QCon London

Tue, 01 Feb 2011 22:58:00 -0800

Hi Folks!

I’ll be giving a 2-part lecture on NoSQL @ Netflix at the Silicon Valley Cloud Computing Meetup in Mountain View on Feb 17. The lectures will be a month apart.

I will detail the challenges involved in going from an RDBMS in our Data Center to AWS’s SimpleDB and S3 in the Cloud. I was intimately involved in this transition. Now, my team at Netflix is investing in Cassandra (and to a lesser extent in HBase). How are we using these different storage options? Come to the meetup to find out!

As a separate note, I am excited to be heading to London in early March to meet engineers in London and to deliver 2 talks

I will also be giving 2 QCon London talks in early March in both Floyd Marinescu’s Architectures You’ve Always Wondered About and Alex Popescu’s NoSQL : Where and How tracks. In the first lecture, I’ll describe Netflix’s Cloud-based data infrastructure. In the second (i.e. the NoSQL Track), I will dive into our NoSQL.

If you happen to be in town, please drop by!

-s

Red Black Trees vs Skip Lists

Wed, 26 Jan 2011 14:50:31 -0800

Red Black Trees vs Skip Lists

How Process Priority Inversion Can Burn CPU via "Waiting" Processes

Tue, 07 Dec 2010 01:32:00 -0800

For the past few weeks, we have been wrestling with an interesting bug in Oracle 11g at Netflix.

We are seeing high CPU attributed with a high number of wait events for the following:

cursor: mutex S
latch: shared pool

I was perplexed as to why waits would result in high CPU so I hopped on to Google. I didn’t find an answer for 11g, but I did find something reported in 10.2 and reportedly fixed in 11g.

In Oracle 10.2, on Operating Systems (e.g. AIX, HPUX, etc..) that support process priority decay (e.g. fair round robin, default on AIX 5), priority inversion can cause “blocked threads” to burn CPU. Ignore the fact that this mentions “cursor pin s” waits instead of “cursor mutex s” waits, the pattern is the same.

http://blog.tanelpoder.com/2010/04/21/cursor-pin-s-waits-sporadic-cpu-spikes-and-systematic-troubleshooting/

In Oracle 10.2, latches cause waits, while mutexes cause CPU spins.

Essentially, there is 1 thread with exclusive access to the cursor mutex and there are N threads requesting shared access to the cursor mutex. All are on the runqueue. The N mutex requester threads each jump onto the CPU and realize that the mutex is not yet free. Instead of going to sleep (i.e. joining the waitqueue), they jump onto the end of the runqueue. This burns CPU. For comparison, if a latch were used, the latch requesters would jump onto the waitqueue and sleep, hence not burn CPU.

The mutex holder can then suffer Unix priority decay or what is known as priority inversion depending on the OS scheduling policy in use. This causes the mutex holder to get less time on the CPU. As a result, the mutex holder cannot release the mutex in a timely manner and the mutex requesters are just busy burning CPU.

This pattern will cause high CPU and “blocked” sessions. According to the article above, this is not an issue on 11g.

-s

Cassandra : Row Cache & Memtable Q&A

Wed, 10 Nov 2010 18:14:00 -0800

Below, you will find 2 sets of questions and answers regarding Cassandra’s (now v0.7) Row Cache and Memtable. These were answered by Matthew Dennis at Riptano, a company that is actively developing both Cassandra and a management suite known as RipCord.

Questions

      Hi!

      I have a super column family. Writes either modify a column within a super column or add a super column to the super column family. I wonder how the row cache works.

      A. How do writes interact with the Row Cache entries?
      1. Do you update the row cache entry in place in order to keep it consistent with the SSTable?
      2. Do you update it when the memtable is updated or when the memtable is flushed to the SSTable?

      B. Also, is the Memtable appended to an SSTable when it is flushed or does it become its own SSTable?

Answers

      The row cache is kept up to date with the memtable in the sense that if the memtable has an entry for the row in it, it is modified to contain the newly written value. If the row cache does not already have the row, it is not added. Getting a row into the row cache requires that the row is read.

      The row cache entry is updated in memory to keep it in sync with the memtable, not the SSTable. If the value is in the row cache, the SSTable(s) are not checked. If the row cache lacks an entry for the row, it is read from the SSTables (possibly by way of the key cache) and added to the row cache if it is enabled.

      It is updated when the memtable is updated, not when it is flushed. The memtable and row cache essentially store a reference to the same row object.

      A memtable is not appended to any SSTable (they are immutable). Each memtable flush creates a new SSTable. Compaction joins multiple SSTables into one larger SSTable.

      On a related note, I generally recommend that people run with row cache disabled.

      1) The OS page cache stores the rows in a more compact way.

      2) Using the OS page cache greatly reduces the GC pressure in the JVM.

      3) Disabling the row cache lowers the load on the server during writes where none of the writes are in the row cache (a very common scenario) because it avoids the check the cache, read the row, insert into the cache, evict something old from the cache path. Not to mention, constantly evicting things from the cache because new things are added only addes to the GC pressure.

      4) Large rows can kill your performance by forcing the JVM to keep it in memory and in extreme cases of multiple large rows and/or *really* large rows can OOM you JVM.

      5) The best case performance improvement you can hope of is generally < 5% and it easily turns negative in most cases.

      The main downside of turning it off entirely is that in the case of rows that are appended to and/or have columns that are overwritten slow enough that pieces of the row make it into different SSTables all the time, it’s important that compaction keeps up.

Questions

      Thanks a lot for this great explanation. We will disable it for now.

      BTW, regarding the row cache eviction scheme, does it use LRU-by-access eviction both
      1. When the row cache is full (i.e. evicts least recently accessed entry when the cache is full)
      2. And by evicting based an age threshold in a separate reaper thread?

      FYI.. by “LRU-by-access” I mean that it’s the least recently accessed row. Alternately, “LRU-by-creation” refers to least recently added to the cache.

Answers

It uses the SECOND_CHANCE eviction policy, which is based on insertion order but an entry gets “a second chance” if it was recently accessed.

That being said, you’re likely interested in https://issues.apache.org/jira/browse/CASSANDRA-975