Web Scalability & Performance: Real Life Lessons

Following is a presentation that I made at TechWeekend in Pune on 5th September. About sixty hard-core technical geeks were present at the sessions. Following is the presentation that I made. Feel free to share.Web Scalability & Performance
You can reach me on Twitter @mukulneetika .

Early YouTube Engineer talks about outages, system scalability

Gigaom.com has posted a very interesting presentation by Cuong Do, an early software engineer who’s now manager of the site’s Core Product Engineering group.

Gigaom.com has a video, here is the transcript of that video.

Introduction

Do’s talk was titled “Behind the Scenes: A Look Into YouTube’s Infrastructure,” had harrowing tales of outages; gory details about the specific languages, architectures, and tools YouTube uses. “One of the key phrases we had in the early days was ‘These are good problems to have,’” Do said. “And after a while we’re like, ‘I’m going to kill the next person who says that.’”

Do describes the “Early team”

• 2 sys admin
• 2 scalability software architects
• 2 feature developers
• 2 network engineers
• 1 DB admin
• Zero chefs

Algorithm for handling rapid growth

while (true) {
identify_and_fix_bottlenecks();
drink();
sleep(TOO_LITTLE);
notice_new_bottleneck();
}

Web Request flow

• End user browser
• NetScalar load balancer
  
• Web Servers (bank of)
  
• Apache
• Local app server
• Python app
• Memcached
• DB

Video served through

• CDN - internal project from Google (for non-US content)
• Video – colo servers – from various US locations

Key technologies

• Linux (SuSE 10.x)
• Apache (2.2.x) / lighthttpd 1.4.x
• lighthttpd - very fast in handling large files
• MySQL 5.0.x - metadata storage
• Python
• [Very difficult to recruit people for a small company, Faster to get 10 machines, than 1 dev, so got more machines that run slow code.]
• Google technologies - search, Bigtable (video thumbnails), GFS (internal storage, transfer buffers)

DB

Started with 1 DB server

• Replica for backup, replica for reporting (later)
• Vertical partitions for DB, takes part of the web site (more later)
• Multiple users for DB, for scalability
• Associate one user with one partition

Scalability challenges

• Rapid unpredictable growth (user growth always exceeds any amount of hardware scale)
• Passionate users (users who video-blog their life)
• New features (recommendation algorithms, compatibility, social graphs – always blew off scalability predictions)
• Pushing hardware and software boundaries (if you are running a hardware or software too close to its limits, it's more likely to fail)
• Unknown problems (issues that you don't find on Google search, or issues that even the vendor of the 3rd-party-software doesn’t know)

One example of an issue we had

Subject: oh @#!%
Date: October 22, 2005 2:24:33 AM PDT
We can't accept any more videos, too many videos.

• All thumbnails were stored in separate sub-directories, and all sub-directories were in one flat directory.
• More than 10K files in a directory, out of Inodes problem.
• Wasn't too difficult to solve, went to a tree structure.
• Wasn't the funniest thing to do, with all videos being uploaded

YouTube: 5 hours Outage

• MySQL gave error, checksum has failed
• Checksum stored for every page (15K of data)
• MySQL checksum failed, it puked, lost 4-5 hours of data
• Took 4-5 hours to recover
• We found lots of questions, but no answers for this problem
• "Maybe this is a hardware issue, not everybody is having it"
• Found exactly one-blog-post, turns out the combination of the RAID card and the other I/O card, can sometimes cause interesting fluctuations in voltage
• So the data was fine in the disk, the voltage fluctuation caused the CPU to read different data
• It took weeks to figure this out

One of most favorite ones: Again ran out of disk space!!

Awesome.

Disk storage - where are we headed?

Some insightful articles and some of my own thoughts on the trends in data storage:

THE BACKGROUND:
Disk capacities are going up and costs are going down, however the effective transfer bandwidth (ETB) per byte of capacity has come down tremendously. Despite capacities and transfer rates increasing by factors or 10,000 and 100 respectively, typical drive ETB has actually decreased by a factor of 100. As Jim Gray said "Disks have become tapes." (Link to source).

Consider, for example, a 10 TB database. Ten years ago, this database would have occupied two thousand 5 GB drives - a common size at the time. With a 3 MB/second transfer rate, the aggregate bandwidth of these 2,000 drives would have been 6 GB/second, enabling the entire database to be scanned in about 30 minutes. Today, only about 20 higher-capacity drives would be needed to hold this same database. Those 20 drives would have an aggregate bandwidth of 1.2 GB/second, increasing the time required to scan the entire database to 150 minutes - an increase of two hours.

DISKS ARE BECOMING A SEQUENTIAL ACCESS DEVICE RATHER THAN A RANDOM ACCESS DEVICE
Jim Gray points out - We have to convert from random disk access to sequential access patterns. Disks will give you 200 accesses per second, so if you read a few kilobytes in each access, you're in the megabyte-per-second realm, and it will take a year to read a 20-terabyte disk. If you go to sequential access of larger chunks of the disk, you will get 500 times more bandwidth—you can read or write the disk in a day. So programmers have to start thinking of the disk as a sequential device rather than a random access device.

Tom White later says that - "MapReduce is a programming model for processing vast amounts of data. One of the reasons that it works so well is because it exploits a sweet spot of modern disk drive technology trends. In essence MapReduce works by repeatedly sorting and merging data that is streamed to and from disk at the transfer rate of the disk. Contrast this to accessing data from a relational database that operates at the seek rate of the disk (seeking is the process of moving the disk's head to a particular place on the disk to read or write data). Read more here.

My take is that SSDs are going to take a while to become an economically viable alternative to disks. Flash disks cost approximately $10/GB, and the OEM costs of good flash drives cost about $60/GB or more (source here). Compare this with the cost of disk, which is about $0.20/GB. So, we are looking at about 300x price difference here. So, I think, it's going to take while before SSDs become reality in storing terabytes of data. Until that time, we will have to use 50-70% empty disks to enhance striping-performance. So, if we were to use 50% empty disks, the cost of disks doubles for storing the same amount of data.

MySQL - Is there a theoretical limit?

Guy Kawasaki interviewed Marten Mickos. Marten was the CEO of MySQL, now he is the senior vice president of the database group within Sun.

Interesting question about MySQL scalability:

Guy: Is there a theoretical limit of MySQL in case a small business uses it and sales/transactions/whatever explode?
Marten: Every software product has its limits, but I think we have shown that MySQL can scale enormously. Google runs its entire ad system on MySQL. Nokia runs mobile phone networks on MySQL. Booking.com runs all their business transactions on MySQL. If a small business reaches those limits, it is not a small business any longer--it is an enormous global player.

I am a fan of MySQL myself.

Good to know that FaceBook also uses MySQL (with it's 65 million users), and YouTube used MySQL and Twitter uses MySQL (Scaling Twitter: Making Twitter 10000 Percent Faster).