Facebook Photo Storage Architecture

Awesome presentation called "Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos", here is an excerpt. You should see the full presentation (if you can get flowgram to work).

Facebook uses MySQL, Memcache, Apache, PHP and Extensions in their application stack.

Facebook uses NetAPP filers for storing files.

Facebook scale of photos

• ~6.5 billion total images, 4-5 sizes stores for each image => ~30 billion files => 540TB total storage capacity.
• ~475,000 images server per second at peak – most through CDNs.
• ~100 million uploaded per week.
  

Facebook uses a 4-tier architecture for serving profiles and photos.

The first tier is CDN, then their proprietary “Cachr”, then their photo servers then the NetApp filers.

Cachr

• “Protects the origin for profile pictures
• Based on modified evhttp
• Uses memcache as backing store
• Microsecond response time on cache hit
• Server can die or restart without losing cache

File handle cache

• Based on lighthttpd
• Uses memcache as backing store
• Reduces metadata workload on NTAP

NetApp storage architectural issues

• NetApp Storage is overwhelmed with metadata
• ~3 disk reads to to read one photo
• Totally bottlenecked on disk bandwidth

Thus, heavy reliance on expensive CDNs to serve reads:

• 99.8% hit rate in CDN for profile images
• ~92% hit rate for photos
• Drastically reduces load on the storage