Saturday, June 28, 2008

Designing Caches for Highly Scalable Web 2.0 Applications

Unix/Linux file systems have been designed in a way that reads are heavily cached and sometimes pre-fetched. There are various techniques, algorithms and methods for read caching, and each file system has its somewhat unique method and therefore performance. Most file systems would use page-cache for caching read-I/O and buffer cache for caching the metadata.

There has been immense amount of research in this area – of how to improve the read performance using caching (see here and here).

Enter highly scalable Web 2.0 era, enter Facebook - if you look at the Facebook IO-profile in my previous post – 92% of the read (for photos) are served by the CDN. What that means is reads will only happen once, after that the file will be cached in the CDN and the read will never go to the backend storage (NetApp filer in this case). So all the file system caching is probably going waste, since we are never going to read from the file-system-cache ever. Facebook photos are cached in CDN for 4.24 years (their http cache-control max-age is 133,721,540), which means the CDN will not go back to the origin server for that period.

This raises interesting questions – do file systems really need to do any caching, what is the read-write ratio for such an application, how can this file system be better tuned for such an application?
Can file system cache be better used for pre-fetching the entire metadata in the cache, so that Facebook NetApp filer has to do fewer than 3 reads for reading a photo?

Thoughts?

Labels: , , , ,

Friday, June 27, 2008

Facebook Photo Storage Architecture

Awesome presentation called "Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos", here is an excerpt. You should see the full presentation (if you can get flowgram to work).

Facebook uses MySQL, Memcache, Apache, PHP and Extensions in their application stack.

Facebook uses NetAPP filers for storing files.

Facebook scale of photos
  • ~6.5 billion total images, 4-5 sizes stores for each image => ~30 billion files => 540TB total storage capacity.
  • ~475,000 images server per second at peak – most through CDNs.
  • ~100 million uploaded per week.
Facebook uses a 4-tier architecture for serving profiles and photos.
The first tier is CDN, then their proprietary “Cachr”, then their photo servers then the NetApp filers.

Cachr
  • “Protects the origin for profile pictures
  • Based on modified evhttp
  • Uses memcache as backing store
    • Microsecond response time on cache hit
    • Server can die or restart without losing cache
File handle cache
  • Based on lighthttpd
  • Uses memcache as backing store
  • Reduces metadata workload on NTAP
NetApp storage architectural issues
  • NetApp Storage is overwhelmed with metadata
  • ~3 disk reads to to read one photo
  • Totally bottlenecked on disk bandwidth
Thus, heavy reliance on expensive CDNs to serve reads:
  • 99.8% hit rate in CDN for profile images
  • ~92% hit rate for photos
  • Drastically reduces load on the storage

Labels: , , ,

Sunday, April 27, 2008

FaceBook server infrastructure

Om Malik reported on Gigaom.com:
The company is running around 10,000 servers, according to Data Center Knowledge, citing comments made by Facebook VP of technology, Jeff Rothschild, at a recent MySQL user conference. (See video of the panel.) Of the 10,000 servers, 1,800 are from MySQL and around 805 of them are memcached servers. In order to house its sprawling infrastructure, Facebook has leased data center space from DuPont Fabros in Ashburn, Va., and Digital Realty Trust in Santa Clara, Calif., DCK reports.

How much is Facebook spending on its infrastructure? The company isn’t going to tell us, but there are clues. Server and storage company Rackable today reported first-quarter 2008 sales of around $69 million. Facebook is one of its largest customers, accounting for around 10 percent of Rackable’s sales (that number could be higher, but we’ll have to wait for Rackable’s 10-Q to get a clearer picture), so some quick, back-of-the-envelope math reveals $7 million in spending by the social networking company. A well placed source of mine just let me know that Facebook is going to spend over $9 million more on servers this year. That should be good news for Rackable. Next on my list is an estimate of Facebook’s bandwidth and data center costs.

Doing a little more calculation - $7M for 10,000 servers, means about $700 per server, assuming a 10% cost for F5,firewall etc. - we are at $630 per server. That's pretty heavy duty server, I think.

Labels: , ,

Tuesday, September 18, 2007

Three kinds of platforms you meet on the Internet - in Mike's words

Please read this insanely useful article written by Mike Anderson.

Mike has described 3 kinds of platforms for web services.

Level 3 platform is the most difficult to build by the platform-provider and is very easy to integrate and use from an end-user’s-perspective. While, Level 1 platform is easiest to build by the platform-provider and difficult to integrate from an end-user’s-perspective. Most people build level-1 platforms – like whenever you see the work “API” it actually means a Level 1 platform. Exception is Facebook and Ning, which are building a Level-2 and Level-3 API respectively. Level 3 is really an API nirvana, and requires very good architectural planning from the platform-provider side. I can imagine some API evolving from a Level 1 to a Level 3; but there aren’t any examples of such APIs; also the code written for Level 1 API, will probably be difficult to use in Level 3, unless there is again some very good architectural planning.

A must read for anybody who codes!

Read on …

Labels: , , , , ,