Sunday, July 27, 2008

Cloud Availability

Cloud Computing has become very widespread with startups as well as divisions of banks, pharmaceuticals companies and other large corporations using them for computing and storage.
Amazon Web Services has led the pack with it's innovation and execution, with services such S3 storage service, EC2 compute cloud, and SimpleDB online database.

Many options exist today for cloud services, for hosting, storage and application hosting. Some examples are below:
Hosting Storage Applications
Amazon EC2 Amazon S3 opSource
MOSSO Nirvanix Google Apps
GoGrid Microsoft Mesh
AppNexus EMC Mozy
Google AppEngine MOSSO CloudFS

[A good compilation of cloud computing is here, with a nice list of providers here. Also worth checking out is this post.]

The high availability of these cloud services becomes more important with some of these companies relying on these services for their critical infrastructure. Recent outages of Amazon S3 (here and here) have raised some important questions such as this - S3 Outage Highlights Fragility of Web Services and this.

[A simple search on can tell you things that you won't find on web pages. Check it out with this search, this and this.]

There has been some discussion on the high availability of cloud services and some possible solutions. For example the following posts - "Strategy: Front S3 with a Caching Proxy" and "Responding to Amazon's S3 outage".

Here I am writing of some thoughts on how these cloud services can be made highly available, by following the traditional path of redundancy.

Cloud Availability configurations The traditional way of using AWS S3 is to use it with AWS EC2 (config #0).

Configurations such as on the left can be made to make your computing and storage not dependent on the same service provider.
Config #1, config #2 and config #3 mix and match some of the more flexible computing services with storage services.
In theory the compute and the storage can be separately replaced by a colo service.

The configurations on the right are examples of providing high availability by making a "hot-standby".

Config #4 makes the storage service hot-standby and config #5 separates the web-service layer from the application layer,
and makes the whole application+storage layer as hot-standby.

A hot-standby requires three things to be configured - rsync, monitoring and switchover.

rsync needs to be configured between hot-standby servers, to make sure that most of the application and data components
are up to date on the online-server. So for example in config #4 one has to rsync 'Amazon S3' to 'Nirvanix' - that's pretty
easy to setup. In fact, if we add more automation, we can "turn-off" a standby server after making sure that the
data-source is synced up. Though that assumes that the server provisioning time is an acceptable downtime,
i.e. the RTO (Recovery time objective) is within acceptable limits.

This also requires that you are monitoring each of the web services. One might have to do service-heartbeating -
this has to be designed for the application, this has to be designed differently for monitoring Tomcat, MySQL,
Apache or their sub-components. In theory it would be nice if a cloud computing service would export APIs,
for example an API for , or
However, most of the times the status page is updated much later after the service goes down. So, that wouldn't help much.
Switchover from the online-server/service to the hot-standby would probably have to be done by hand.
This requires a handshake with the upper layer so that requests stop and start going to the new service
when you trigger the switchover. This might become interesting with stateful-services and also where
you cannot drop any packets, so quiscing may have to be done for the requests before the switchover takes place.

Above are two configurations of multi-tiered web-services, where each service is built on a different cloud service. This is a theoretical configuration, since I don't know of many good cloud services, there are only a few. But this may represent a possible future, where the space
becomes fragmented, with many service providers.

Config #7 is config #6 with hot-standby for each of the service layers.
Again this is a theoretical configuration.

Cost Impact
Any of the hot-standby configurations would have cost impact - adding any extra layer of high-availability immediately adds to the cost, at least doubling the cost of the infrastructure. This cost increase can be reduced by making only those parts of your infrastructure highly-available that affect your business the most. It depends on how much business impact does a downtime cause, and therefore how much money can be spent on the infrastructure.

One of the ways to make the configurations more cost effective is to make them active-active configuration also called a load balanced configuration - these configurations would make use of all the allocated resources and would send traffic to both the servers. This configuration is much more difficult to design - for example if you put the hot-standby-storage in active-active configuration then every "write" (DB insert) must go to both the storage-servers, writes (DB insert) must not complete on any replicas (also called mirrored write consistency).

Cloud Computing becoming mainstream
As cloud computing becomes more mainstream - larger web companies may start using these services, they may put a part of their infrastructure on a compute cloud. For example, I can imagine a cloud dedicated for "data mining" being used by several companies, these may have servers with large HDDs and memory and may specialize in cluster software such as Hadoop.

Lastly I would like to cover my favorite topic -why would I still use services that cost more for my core services instead of using cloud computing?
  1. The most important reason would be 24x7 support. Hosting providers such as servepath and rackspace provide support. When I give a call to the support at 2PM India time, they have a support guy picking up my calls – that’s a great thing. Believe me 24x7 support is a very difficult thing to do.
  2. These hosting providers give me more configurability for RAM/disk/CPU
  3. I can have more control over the network and storage topology of my infrastructure
  4. Point #2 above can give me consistent throughput and latency for I/O access, and network access
  5. These services give me better SLAs
  6. Security

Labels: , , , , , ,

Wednesday, July 23, 2008

Comparing Clouds: Amazon EC2, Google, AppNexus, and GoGrid

InfoWorld has an awesome article published by Peter Wayner, who compares various cloud computing services - Amazon, Google, AppNexus, and GoGrid. Read more here.

Following is an excerpt:

Labels: , , , , , ,

Tuesday, July 22, 2008

Scientists found a workable way of reducing CO2 levels reports - Scientists say they have found a workable way of reducing CO2 levels in the atmosphere by adding lime to seawater. And they think it has the potential to dramatically reverse CO2 accumulation in the atmosphere, reports Cath O'Driscoll in SCI's Chemistry & Industry magazine published today.

Adding lime to seawater increases alkalinity, boosting seawater's ability to absorb CO2 from air and reducing the tendency to release it back again.

The important point is that when you put lime into seawater it absorbs almost twice as much carbon dioxide as is produced by the breaking down of the limestone in the first place.

The Cquestrate project has a web page; here's how they describe the idea; and here's a chance of getting involved. I think this is HUGE!

They also have this awesome slideshow presentation, which goes into the gory but interesting details of all the chemical reactions.

Here is their idea of how they would do it (subject to economic feasibility):
  • The Nullarbor Plain is the world's largest single piece of limestone and occupies an areas of about 200,000 km2
  • With an average thickness of 50m, there is 10,000 km3 of limestone
  • To sequester 1 billion tonnes of carbon (GtC) would require the excavation of 1.5 km3 of limestone
  • Between 1750 and 2004 humankind has emitted 305 GtC. Current emission rates are about 7 GtC per year
  • Thus employing this process on 500 km3 of limestone (about 5% of the limestone in the Nullarbor Plain) would return carbon dioxide levels to pre-industrial levels.

I hope all of this comes true. I think, economic viability of converting CaCO3 into CaO will be the biggest hurdle.

Labels: ,

Sunday, July 13, 2008

Early YouTube Engineer talks about outages, system scalability has posted a very interesting presentation by Cuong Do, an early software engineer who’s now manager of the site’s Core Product Engineering group. has a video, here is the transcript of that video.

Do’s talk was titled “Behind the Scenes: A Look Into YouTube’s Infrastructure,” had harrowing tales of outages; gory details about the specific languages, architectures, and tools YouTube uses. “One of the key phrases we had in the early days was ‘These are good problems to have,’” Do said. “And after a while we’re like, ‘I’m going to kill the next person who says that.’”

Do describes the “Early team”
  • 2 sys admin
  • 2 scalability software architects
  • 2 feature developers
  • 2 network engineers
  • 1 DB admin
  • Zero chefs
Algorithm for handling rapid growth
while (true) {
Web Request flow
  • End user browser
  • NetScalar load balancer
  • Web Servers (bank of)
  • Apache
  • Local app server
  • Python app
  • Memcached
  • DB
Video served through
  • CDN - internal project from Google (for non-US content)
  • Video – colo servers – from various US locations
Key technologies
  • Linux (SuSE 10.x)
  • Apache (2.2.x) / lighthttpd 1.4.x
  • lighthttpd - very fast in handling large files
  • MySQL 5.0.x - metadata storage
  • Python
  • [Very difficult to recruit people for a small company, Faster to get 10 machines, than 1 dev, so got more machines that run slow code.]
  • Google technologies - search, Bigtable (video thumbnails), GFS (internal storage, transfer buffers)
Started with 1 DB server
  • Replica for backup, replica for reporting (later)
  • Vertical partitions for DB, takes part of the web site (more later)
  • Multiple users for DB, for scalability
  • Associate one user with one partition
Scalability challenges
  • Rapid unpredictable growth (user growth always exceeds any amount of hardware scale)
  • Passionate users (users who video-blog their life)
  • New features (recommendation algorithms, compatibility, social graphs – always blew off scalability predictions)
  • Pushing hardware and software boundaries (if you are running a hardware or software too close to its limits, it's more likely to fail)
  • Unknown problems (issues that you don't find on Google search, or issues that even the vendor of the 3rd-party-software doesn’t know)
One example of an issue we had
Subject: oh @#!%
Date: October 22, 2005 2:24:33 AM PDT
We can't accept any more videos, too many videos.
  • All thumbnails were stored in separate sub-directories, and all sub-directories were in one flat directory.
  • More than 10K files in a directory, out of Inodes problem.
  • Wasn't too difficult to solve, went to a tree structure.
  • Wasn't the funniest thing to do, with all videos being uploaded
YouTube: 5 hours Outage
  • MySQL gave error, checksum has failed
  • Checksum stored for every page (15K of data)
  • MySQL checksum failed, it puked, lost 4-5 hours of data
  • Took 4-5 hours to recover
  • We found lots of questions, but no answers for this problem
  • "Maybe this is a hardware issue, not everybody is having it"
  • Found exactly one-blog-post, turns out the combination of the RAID card and the other I/O card, can sometimes cause interesting fluctuations in voltage
  • So the data was fine in the disk, the voltage fluctuation caused the CPU to read different data
  • It took weeks to figure this out
One of most favorite ones: Again ran out of disk space!!


Labels: , , ,

Thursday, July 10, 2008

Best Web 2.0 Sites in India

I am compiling a list of best Web 2.0 sites in India. I am most likely going to do a writeup on each of the sites, and then suggest making a group of like minded people, who can exchange great ideas. The initial list is following in no particular order. Feel free to add more in this list, by commenting this post:

      Labels: , , , , , , ,

      Monday, July 07, 2008

      Google Advanced Search - now with dates!


      Labels: , ,