Practical Cloud Computing

Blogged hacked.. password changed. Sigh!

Tue, 17 Apr 2012 21:59:13 -0700

sharad_ag: Durable Event Data Transport at Scale

Wed, 28 Mar 2012 10:48:11 -0700

sharad_ag: Durable Event Data Transport at Scale:

A lot is spoken about parallel computing, large scale storage and realtime analytics these days. Relatively lot less is said about transporting event data from producer to consumer reliably and at scale. The problem is not unique to any organization dealing with BigData and definitely non-trivial….

myNoSQL: What Is Unique About LinkedIn's Databus

Thu, 22 Mar 2012 11:10:06 -0700

myNoSQL: What Is Unique About LinkedIn's Databus:

nosql:

After learning about LinkedIn’s Databus low latency data transfer system, I’ve had a short chat with Sid Anand focused on understanding what makes Databus unique.

As I’ve mentioned in my post about Databus, Databus looks at first as a data-oriented ESB. But what is innovative…

Jay Kreps: Getting Real About Distributed System Reliability

Mon, 19 Mar 2012 13:37:27 -0700

Jay Kreps: Getting Real About Distributed System Reliability:

boredandroid:

There is a lot of hype around distributed data systems, some of it justified. It’s true that the internet has centralized a lot of computation onto services like Google, Facebook, Twitter, LinkedIn (my own employer), and other large web sites. It’s true that this centralization puts a lot of…

Who Really Uses LinkedIn (Infographic)

Sat, 03 Mar 2012 22:03:34 -0800

Who Really Uses LinkedIn (Infographic)

Database Isolation Levels (MySQL)

Thu, 16 Feb 2012 23:07:00 -0800

This content below was part of my answer to a question on Quora about MySQL Isolation levels: http://b.qr.ae/xhWSbP

Answer from Quora

Here are some useful links:

Notes:

In the second link, have a look at the “Read phenomena” section

To understand these isolation levels, imagine 2 transactions:

Imagine that the table FOO has one record {name:’sid’, age:35} to start with. The first transaction is doing 2 reads while the second transaction is updating the age from 35 to 36.

Transaction_1

Start transaction

Select * from FOO where name=’sid’

Select * from FOO where name=’sid’

commit

Transaction_2

Start transaction

Update FOO set age = 36 where name = ‘sid’

commit

I try to show that Transaction_1’s 2 selects envelop Transaction_2’s update.

Ok, so now let’s look at what happens under each of the 4 isolation levels:

Read Uncommitted : Transaction_1’s first select will see ‘sid’ with an age of 35 but its second select will see ‘sid’ with an age of 36 . This is true even if Transaction_2’s commit failed and rolled back. The fact that Transaction_1’s second select read data that was never committed is called a “dirty read”. This is horrible.

Read Committed : Transaction_1’s first select will see ‘sid’ with an age of 35 but its second select will ‘sid’ with an age of 36 if and only if Transaction_2’s commit succeeded. The fact that Transaction_1’s two selects are seeing the same row with different data (i.e. age changed) is called a non-repeatable read.

Repeatable Read : The phenomena above cannot happen. However if Transaction_2 were altered to insert a record with {name:’sid’, age:100} after it executed the insert, then Transaction_1 would see two rows on its second select, one for “sid” with age=35 and another with age=100. In other words, between the 2 selects in Transaction_1, a new “phantom” row appeared. This is called a “phantom read”.

Serializable : “Phantom reads” cannot occur.

The State of NoSQL in 2012

Thu, 19 Jan 2012 00:47:00 -0800

Preamble Ramble

If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”.

In 2007, Amazon published a paper on Dynamo. The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud.

It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing.

Huh?

A month into the investigation, we start wondering about our Oracle database. How are we are going to move it into the cloud? That’s when we are told that we are to abandon our RDBMS too. Instead, we are going to focus on “AP”-optimized systems. “AP” or high-availability systems is our focus in 2008-2010 as Netflix launches its video streaming business. What’s more available than TV right? Hence, no downtime is allowed, no excuses!

Fast forward to the end of 2011: the past 3 years have been an amazing ride. I helped Netflix with two migrations in 2010: the first was from Netflix’s datacenter to AWS’ cloud, the second from Oracle to SimpleDB. 2011 was no less exciting: with a move from SimpleDB to Cassandra, Netflix was able to expand overseas to the UK and Ireland. Since Cassandra offers configurable routing, a cluster can be spread across multiple continents.

Fast forward to today: I’ve completed my first month at LinkedIn. I’ve spent this month getting familiar with the various NoSQL systems that LinkedIn has built in-house. These include Voldemort (another Dynamo-based system), Krati (a single-machine data store), Espresso (a new system being actively developed), etc… LinkedIn is now facing similar challenges to the Netflix of 3 years ago. Traffic is growing and both Oracle and the datacenter face potential obsolescence.

NoSQL and Cloud Computing to the rescue?

The State of NoSQL Today

Ignoring the datacenter vs. public cloud question for the time-being, what would I pick today regarding a NoSQL alternative to Oracle? Not many people get a chance to solve the same tough problem twice.

For one, there are still quite a few NoSQL alternatives in the market. Some are supported by startups (e.g. Cassandra, Riak, MongoDB, etc..), some are supported indirectly by companies (e.g. LinkedIn’s support of Voldemort, Facebook & StumbleUpon’s support of HBase), and some are supported directly by companies (e.g. AWS’s S3, SimpleDB, and DynamoDB, etc… ).

In making a decision, I’ll consider the following learnings:

Any system that you pick will require 24-7 operational support. If it is not hosted (e.g. by AWS), be prepared to hire a fleet of ops folks to support it yourself. If you don’t have the manpower, I recommend AWS’ DynamoDB
Just because the company got by with one big machine for Oracle, don’t be surprised if the equivalent NoSQL option results in 36 smaller machines. All complete solutions to fault-tolerance support “rebalancing”. Rebalancing speed is determined by data size of a shard. Hence, it’s better to keep the size per shard reasonable to minimize MTTR in times of disaster.
Understand the limitations of your choice:
- MongoDB, at the time of this writing, has a global write-lock. This means that only one write can proceed at a time in a node. If you require high write-throughput, consider something else
- Cassandra (similar to other Dynamo systems) offers great primary key-based access operations (e.g. get, put, delete), but doesn’t scale well for secondary-index lookups
- Cassandra, like some other systems, has a lot of tunables and a lot of internal processes. You are better off turning off some features (e.g. Anti-entropy repair, row cache, etc…) in production to safeguard consistent performance.

Many of the NoSQL vendors view the “battle of NoSQL” to be akin to the RDBMS battle of the 80s, a winner-take-all battle. In the NoSQL world, it is by no means a winner-take-all battle. Distributed Systems are about compromises.

A distributed system picks specific design elements in order to perform well at certain operations. These design elements comprise the DNA of the system. As a result, the system will perform poorly at other operations. In the rush to win mindshare, some NoSQL vendors have added features that don’t make sense for the DNA of the system.

I’ll cite an example here. Systems that shard data based on a primary key will do well when routed by that key. When routed by a secondary key, the system will need to “spray” a query across all shards. If one of the shards is experiencing high latency, the system will return either no results or incomplete (i.e. inconsistent) results. For this reason, it would make sense to store the secondary index on an unsharded (but replicated) system. This concept has been utilized internally at Netflix to support internal use-cases. Secondary indexes are stored in Lucene to point to data in Cassandra.

LinkedIn is following the same pattern in the design of its new system, Espresso. Secondary indexes will be served by Lucene. The secondary index will return rowids in the primary store.

This brings me to another observation. In reviewing Voldemort code recently, I was impressed by the clarity and quality of the code. In core systems, whether distributed or not, code quality and clarity goes a long way. Although Voldemort is an open source project and has been for years, a high degree of discipline has been maintained. I was also impressed by the simplicity of its contract - get(key), put(key,value), and delete(value). The implementors understood the DNA of the system and did not add functionality to the system that is ill-suited to the DNA.

In a similar vein, Krati, Kafka, Zookeeper, and a few other notable open-source projects stick to clear design principles and simple contracts. As such, they become reusable infrastructure pieces that can be used to build an Distributed System that you need. Hence, the system we end up building might be composed of several specialty component systems that can be independently tuned, in some ways similar to HBase. As a counter example, to achieve predictable performance in Cassandra without a significant investment in tuning, it may be easier to turn off features. This is because multiple features in a single machine contend for resources — since each feature has a different DNA (or resource consumption profile), performance diagnosis and tuning can be a pain point.

Till next time, stay tuned!

Netflix & LinkedIn

Sat, 10 Dec 2011 16:52:00 -0800

After 4+ years at Netflix, I am about to embark on a new challenge. On Monday, I start at LinkedIn as a senior member of the Service Infrastructure team. It has been my privilege and honor to work at Netflix these many years. To date, Netflix is the best company that I have worked at.

Netflix

Central to the company’s success is its unique culture of “Freedom and Responsibility”. This culture places decision making, risk taking, innovation, and operational judgement in the hands of each engineer. Management serves to prioritize the big tasks and sets the context for each engineer, leaving each engineer to prioritize his or her own work. There is an allergic aversion to process and micromanagement.

Progress is made by sheer drive and motivation. Accountability is achieved by having each engineer accountable for his or her own work. Gaps are filled not by instituting process but by leveraging automation (e.g. our Simian Army).

This unique culture attracts and rewards folks who are “do-ers not say-ers” and fully-formed adults who play nice with others. In other words, “brilliant jerks” tend to have a short shelf life at Netflix, whereas someone with a good blend of leadership, technical savvy, and social intelligence will go far.

LinkedIn

Frankly, it is this culture and my relationships with cherished colleagues that have kept me from considering positions at other companies for so long.

Fast forward to my acceptance of a position at LinkedIn. If Netflix were to have a peer in terms of technical vision, engineering drive, and a bright future, it would be LinkedIn. We all know LinkedIn as the place to find jobs or employees, manage professional contacts, and follow technology trends, but I believe that feature set to be the tip of the iceberg. Speaking to the technical attractiveness of the company, LinkedIn has long been a leader in big data and analytics, open sourcing technology with some regularity. With renewed post-IPO vigor and an infusion of new talent, they’ve bolstered their commitment to open source and technology evangelism.

As a new entry to their Service Infrastructure team, I hope to share my experience regarding the following:

the development and evangelism of centralized libraries and services
the transition to high availability databases
best practices around high availability and high scalability

A large part of my role at Netflix involved the migration to a public cloud, but it is not clear whether this is something that LinkedIn finds useful.

I look forward to working with Kevin Scott, Mohak Shroff, and their talented teams.

Slides from my QCon SF 2011 talk on Fault Tolerant Systems at Netflix

Sun, 13 Nov 2011 18:49:00 -0800

Abstract : http://qconsf.com/sf2011/speaker/Siddharth+Anand

Keeping Movies Running Amid Thunderstorms!

View more presentations from Sid Anand

"Head in the Cloud" Article about Netflix in the Cloud

Thu, 03 Nov 2011 18:59:00 -0700

Check out the latest Cornell Engineering Magazine Issue for an article on how Netflix leverages the cloud! The story starts on page 10.

Full Version of “The a la menthe” Translation of...

Fri, 09 Sep 2011 00:09:36 -0700

Full Version of “The a la menthe”

Translation of Lyrics : http://www.williamsonday.com/morocco/archives/2007/02/boy-i-remember.html

"NoSQL @ Netflix, Part 2" Talk at OSCON 2011

Wed, 27 Jul 2011 10:01:52 -0700

I’ve been having a great week so far at OSCON. Excellent talks from Riak, Facebook, 10gen, DataStax, Acunu, StumbleUpon, etc….

Uploading slides from my talk here..

OSCON Data 2011 — NoSQL @ Netflix, Part 2 View more presentations from Sid Anand

NoSQL @ Netfix @ OSCON Data, July 25

Sun, 10 Jul 2011 18:31:34 -0700

During July 25-27, O’Reilly will be dedicating a special section of their OSCON open source conference to big data use-cases and technologies. Tracks cover a range of topics, from NoSQL to relational technologies. Additionally tracks dedicated to Hadoop and Visualization are also included. This section is aptly named OSCON Data.

I’ll be speaking during the Monday, 10:40 am slot about Netflix Data Architecture in the cloud. Specifically, I’ll cover how we moved from Oracle to a range of cloud based data storage, including SimpleDB, S3, & Cassandra. Our experience with Cassandra is still in the formative stage, so I will cover our learnings and future plans for the technology.

Be sure to check out my talk

Sid Anand, Cloud Data Architect, Netflix

Epidemics and Gummy Bears

Wed, 06 Jul 2011 14:13:51 -0700

Epidemics and Gummy Bears

Acunu - Cassandra optimized (SSDs, new data structure)

Mon, 27 Jun 2011 22:14:00 -0700

]

For some workloads, they achieve 2 orders of magnitude better latency results relative to a vanilla Cassandra distribution. This is achieved through a combination of SSDs and a new data structure (Stratified B-tree).

Acunu White paper. http://bit.ly/jQc423

Short Blog entries on Cassandra performance

http://bit.ly/mUqWLO Cassandra under heavy write load, Part 1

http://bit.ly/mMT0Ts Cassandra under heavy write load, Part 2

Remembering B-trees http://bit.ly/lH8fwh

http://bit.ly/lryYbf Copy-on-Write B-tree beaten by Stratified B-tree

http://bit.ly/iJg01H A summary of the above paper

http://bit.ly/jV5j1x More on why Fractal-tree DBs scale better on SSDs. Cache oblivious Fractal Trees

http://slidesha.re/jIRkYj More on streaming data structures : doubling array, fractional cascading, and stratified doubling arrays explained (presentation) <—- V. Good

Recent GigaOM Structure 2011 Cloud Guru Panel Video (6/22/2011)

Thu, 23 Jun 2011 13:20:35 -0700

Recent GigaOM Structure 2011 Cloud Guru Panel Video (6/22/2011)

Netflix @ DataStax SF 2011 - Monday, July 11

Mon, 13 Jun 2011 11:26:00 -0700

Netflix will be sponsoring DataStax SF 2011 on Monday, July 11. Though Netflix does not often sponsor events or recruit at conferences, we hope to engage with others active in the Cassandra and Distributed Systems community.

Stop by our booth to learn about what we are working on! Bring your resume as well!

Sid, Cloud Systems, Netflix

Some upcoming conferences that I will be speaking at ...

Fri, 03 Jun 2011 01:25:00 -0700

Netflix for Mobile in the Cloud

Tue, 31 May 2011 22:08:00 -0700

Last week, I attended Intuit’s annual CTOF (Create The OFfering) conference in San Diego. The focus this year was on mobile best practices. I was happy to represent Netflix at this venue and learn from other companies with a foot in either mobile, the cloud, or both.

Netflix is an example of both with support on several mobile platforms (e.g. iOS, Android, Windows Mobile) and traffic served from Amazon’s Cloud.

Intuit CTOF 2011 - Netflix for Mobile in the Cloud View more presentations from Sid Anand

If you are interested in positions at Netflix, working on either the Cloud or mobile, we are hiring!

If you are just stopping by for your Calvin and Hobbes fix, enjoy!

Distributed Systems & Dev Ops Engineers, Netflix Wants You!

Mon, 23 May 2011 16:57:00 -0700

Hi Folks!

As many of you know, Netflix has been expanding its video streaming offering throughout the US and the world. We have been successful thanks to our move to Amazon Web Services’ cloud and our adoption of NoSQL solutions.

-s