Do We Need Cloud API Standards?
I was on a panel entitled “HYBRID CLOUDS — THE BEST OF BOTH WORLDS?” at the Structure 2010 conference this past week. (Check out a recording of the panel) For those readers who are unfamiliar with the concept of the “hybrid” cloud, essentially, it’s the idea that one logical infrastructure cloud can be composed from two or more clouds of different deployment locale (on-premises vs. elsewhere) or visibility levels (public vs. private)
I was lucky enough to be on this panel with a number of very intelligent people who had some amazing insight into the topic. One of the questions posed to the panel really struck a chord. Essentially, the question was something along the lines of “Are standards necessary for the hybrid cloud to become a reality?” (Roughly minute 29:07 in the recording) The standards being referred to are standards that would abstract the command and control of virtual infrastructure. Each cloud provider (e.g., Amazon through EC2, GoGrid, etc.) typically provides an API by which one can control the deployment and ongoing operations of a virtual resource. Since each cloud provider has a different API, there is a form of lock-in that rears its head since, although each cloud might provide a similar container (say, a Windows 2008 Server), your code for managing and leveraging that cloud is non-portable.
Mårten Mickos, CEO of Eucalyptus, fielded the question stating that standards are necessary for hybrid cloud to really take off and that a standard would accelerate adoption significantly. I agree with the spirit behind Mårten’s view, but not with the idea of formal standards (and as you’ll read, I think Eucalyptus is poised to do some amazing things). Now when we discuss “standards” we mean an industry standard sponsored by a standards body or some sort of consortium. Others on the panel respectfully disagreed with Mårten’s position. Joe Weinman responded that standards aren’t necessary, and that different providers need to be free to evolve solutions that best fit the problem domain and customer need. My position was somewhat in alignment with Joe’s; standards, by definition, must cater to lowest common denominator subsystems which can bias efforts towards standardization rather than on new techniques to solve new problems in the cloud. It’s important to understand why this is the case, and also why it doesn’t mean that although a formal standard shouldn’t be pursued in the near term, generalization should absolutely be pursued.
In some instances, “formal” standards can stifle innovation or distract focus from solving specific problems by becoming abstract enough to be standards compliant, but not specific enough to solve a problem as well as possible. For example, if some cloud provider comes up with a novel way to address something new, they can still conform to the standard, but any uniqueness they bring to the table will be lost behind the abstraction. Just think of the SQL-92 standard. Most every DB vendor has some level of conformity, but various RDBMS’ do something amazingly useful things outside of the SQL-92 standard. If you leverage those proprietary pieces, you can no longer benefit (at least not by much) from the standard itself since you’ve broken conformity. Furthermore, any standards that have evolved in the past have typically fragmented into many sub-standards, dialects, etc. (just like the SQL case)
Now, does this mean that we shouldn’t pursue standardization? Well, if we’re talking about a “formal” standard, we probably don’t need to rush it. Technological generalization, however, is a different story. We’re at a point in the cloud’s evolution that requires organizations like Eucalyptus to focus on providing a coherent means to manage multiple specific cloud implementations. This is amazingly powerful and valuable and doesn’t require a formal standard. Eucalyptus has focused generalizing the Amazon EC2 API as a standard API that could be applied against any other cloud system. Overall, EC2 is a wise choice since it has the broadest market penetration and the most generally applicable API when looking from an abstraction point of view. Eucalyptus is doing some amazing work to make sure that you can leverage API expectations to work with a variety of virtualization clouds. Their efforts will undoubtedly boost adoptability, reduce adoption risk, and increase the viability of private and hybrid cloud deployments. Given Amazon’s API penetration, a market “gold standard” has been created just by shear adoption, and I think people will naturally flock to a proper abstraction like Eucalyptus because of sheer necessity to ensure compatibility with the gold standard while preserving the ability to experiment with new solutions. By choosing EC2, Eucalyptus’ approach provides a standards-like advantage without the baggage since the standard is chasing the market leader, rather than creating a standard and trying to enforce it. Kudos to Mårten and Eucalyptus for helping catalyze adoption of the cloud!
Do you feel that formal API standardization will be absolutely required in order for cloud, private cloud, or hybrid cloud to work, or will things organically tend to a relatively standard oligopoly of well known abstractions without a formal standard in place? Do standards create stifle and slow down innovation by always focusing on the lowest common denominator, or does it promote innovation by guaranteeing a baseline?
Related Posts
Tags
Share This
Similar questions are being asked around the SaaS accounting software space we (KashFlow) operate it. Should we expose accounting data in a standard format and have standardised APIs?
I don’t see it happening in our space or in the general cloud (or PaaS) area any time soon just because it’s not in the commercial interests of the bigger players to do so.
Perhaps there’s a business opportunity for someone to provide a proxy/abstraction layer for PaaS. It receives commands in one format and then passes them on to the relevant provider in the format they expect.
It adds no value whatsoever and could even be seen as an additional point of failure in a system.
But on the plus side it future-proofs your operations. If you want to change vendors (either permanently or temporarily due to your first vendor having problems) you simply login to your proxy account and change your settings so you use the new vendor. No changes required in your own code.
I found a great video on this topic which gives a framework of questions to evaluate if you should go to the Cloud and how to assess whether to have a public cloud, a private cloud or a hybrid solution [regardless of who is providing the cloud i.e. Amazon, SalesForce, Windows Azure, etc.]
“Bridging the Gap from On-Premises to the Cloud” by Yousef Khalidi:
http://microsoftpdc.com/Sessions/SVC20
thoughts?
hope that helps,
-cn
I think standards arent necessary for the hybrid cloud at this point, to give it a room to evolve. But I believe at some point there should be some form of standard to give a more structured frame work.
Do standards not stifle creativity to a point? Or am I way off base as usual? Standards are adopted after something is established or at least way into the process. ??
Great thoughts Sinclair. This is a very interesting conversation, but a huge topic; thus just some comments here trying to be not too verbose.
To your question, my personal opinion is that “formal API standardization” is not “absolutely required”. Philosophically I’m with the “innovate now, standardize later” camp as I think the trade-offs still favor innovation over standardization in this area today, plus the rest of the IT world still operates in that mode, thus would cloud computing have a better chance at standardization?
Fundamentally though, I think we could ask the question a little differently. Instead of applying that question to cloud computing as a whole, it might make more sense, and more feasible, to look at certain areas/layers in cloud computing as places where standardization may add more value than constraints.
At a high-level, the industry is differentiating between infrastructure, platform, and software as-a-service offerings (i.e., IaaS, PaaS, SaaS). At this moment, specialization levels increase significantly as we move up the stack. Public PaaS offerings such as Windows Azure, AWS, App Engine, Force.com, etc., are already more different than similar, and the differentiations grow as we get into SaaS, and then into information management, and so on. The opportunity for standardization is really only available at the lower levels in IaaS offerings, as there is more commonality and established standards and processes in terms of how customers operate and manage infrastructure. For example, a lot of focus today is to support cloud federation to provide elasticity for private clouds, but that’s just one abstraction layer on top of provisioning and managing VM’s (over-simplifying a bit here).
However, at the same time, why standardize when, as you and others have pointed out, companies like Eucalyptus can help mitigate and manage the differences in underlying API’s and providing that abstraction at a certain level? After all, cloud-as-a-platform provides opportunities for to build layers and layers of abstractions to add value in different ways. Also in a way, this is where cloud computing and traditional on-premise software operations differ, fundamentally. Cloud computing inherently allows us to work in a dynamic environment, where changes can be more frequent, and in fact preferred. On the other hand, on-premise software operations today tend to be more on the static side of things, and standardization helps to manage and mitigate changes and differences when we have a heterogeneous infrastructure to operate.
Thus standardization can be considered an established approach to help us better manage the on-premise world. From this perspective, is it necessary or beneficial to try to enforce this particular traditional approach to a different paradigm? That is of course, if we think cloud computing represents a new paradigm even though it’s built upon existing technologies and best practices. Personally I think cloud computing represents something different than just trying to host VMs in different places, and more benefits can be gained by leveraging it as a new paradigm (and that’s a whole other topic to dive into). :)
Just my thoughts. Best, -David Chou (Microsoft)
Duane: good point on both the business opportunity as well as the future proofing. I think when people are adopting something new like the cloud, anything to mitigate risk goes a long way in reducing/removing adoption hurdles.
Surge: Standards definitely can stifle creativity early on. I think standards have an appropriate place in the lifecycle of anything, but it tends to be at maturity when the dust settles a bit.
David: Excellent thoughts! I think the takeaway that the lower parts of the “cloud stack” are more standardization friendly is key, particularly since it represents the fundamental building blocks for everything else in the cloud.
On your notion that
“…cloud computing represents something different than just trying to host VMs in different places, and more benefits can be gained by leveraging it as a new paradigm”
- I absolutely agree! Rather than “cloud washing” everything, understanding that a new paradigm exists and leveraging as such is a catalyst to value creation. I think it takes new technologies like PaaS plays and cloud middleware plays like SaaSGrid to help individuals access new paradigms without the learning curve or cost. Good fodder for a new post;-)
To answer the question we just have to look at relational databases as a model.
The ANSI SQL standard tremendously increased the rate of adoption of RDBMS technology, yet did nothing to prevent database vendors like Oracle and Microsoft from adding proprietary extensions to their product offerings.
Developers have (and make) a choice whether or not to trade portability for non-standard features. It’s a well understood compromise.
Cloud vendors can do the same thing: support the Cloud standard so that developers who want portability can have a core API to enable that and also differentiate or specialize their offerings for those who don’t mind trading portability for extended functionality.
What’s so hard about that?
TJL
Thomas, spot on with the ANSI SQL references. I think my only difference with your perspective is timing. SQL-86 in 1986 was the first ANSI adopted standard, which was almost 15 years after Chamberlin developed SQL and was well after some players such as Oracle and IBM had made SQL commercially available for almost half a decade.
Some commercial maturing occurred that helped “shake-out” what the ANSI SQL standard should be. I think if the ANSI standard was developed before truly evidencing commercial viability of what was needed in SQL, a standard would have hindered adoption rather than boosted.
In the same spirit, I think a standard will be warranted in the cloud, but after the market has matured enough to provide indicators for what that standard should look like. IMHO, we’re just not there yet.