Lack of Multi-tenancy: A Broken Business Architecture?

Nov 4, 2010 by Sinclair Schuller

A couple of months back, there was a rekindling of good debate around multi-tenancy, it’s importance (or lack thereof), and who benefits most (if at all) from multi-tenancy. Cost angles as well as soft dollar angles were discussed, as was a good overview of the importance of multi-tenancy being represented in the aggregation of data into a single, digestible source and the value that creates all on its own.

For anyone who has read SaaSBlogs before, you probably know that I’m a passionate supporter of single instance, multi-tenancy as the architecture of choice in most situations (there are some good edge cases). Clearly, I side with the proponents of multi-tenancy in most of these debates, but I want to tackle something different this time. Typically, the supporting evidence is anchored on cost savings and efficiency that multi-tenancy brings to the table. Let’s look at the multi-tenancy discussion through a different lens: that of a business architecture.

We rarely consider the ramifications of software architecture decisions on fundamental business processes. Most software to date has been developed in a vacuum because the cost of operations and delivery have typically been borne by the customer, not the software company that created the product. I find that SaaS vendors are surprised when I discuss their technical decisions in terms of how it impacts their ability to sell in a most real sense. Let’s start with a simple example: offering free trials or running a beta program.

Free trials are used to provide prospects with some sort of evaluation period of your software so that they may make an informed buying decision. Beta programs give prospects and existing customers early access to new offerings, or to changes in existing offerings existing customers already subscribe to. In the on-premises world, this was a simple thing to do because you shipped bits to customers at no cost to you. As a SaaS provider, things are different: for both of these use cases, the SaaS provider must bear the incremental cost of delivery of each prospect or customer that asks for a free trial or entrance to a beta without associated revenue. The SaaS provider, being responsibility for delivery, now takes on the full operating burden that the customer used to take on to run an on-premises free trial or beta.

Now you ask “Sinclair, so where does multi-tenancy fit in?” If you understand multi-tenancy, you know that it drives up the ratio of the number of customers that fit on a given physical or virtual infrastructure by forcing resource sharing within application stack components. A “tenant” is a meta-construct that does not require physical boundaries for isolation. This means that the incremental cost of acquiring a new customer is as close to $0 as possible, and can remain at $0 if the customer posts little activity. Everything from your Oracle/SQL Server licenses to your physical or virtual infrastructure costs are spread across customers. A single instance per tenant model, even in a virtualized case, is severely broken in this respect. Let’s use a more concrete thought experiment to understand this.

Let’s say that we decided to use Amazon’s EC2 to virtualize our offering to provide for tenant isolation; that is, we create an EC2 virtual machine instance for each customer to our SaaS offering. As of the writing of this post, a small Linux EC2 instance costs $0.085 and a small Windows EC2 instance costs $0.12. Let’s use this and some other assumptions  to build a model to understand what free trials in a non multi-tenant world due to Customer Acquisition Cost (CAC) and the payback period of the CAC. Since Apprenda is in the .NET world, we’ll focus on Windows instances. This means that for every hour of a free trial we run for a single prospect, we pay $0.12. Offering a 1 month trial is quite typical. Now let’s assume that a customer earns us $250.00 in monthly revenue (for the sake of realism, maybe we earn $50.00 per seat per month, and the average deal size is 5 seats). Furthermore, let’s also assume that for every 20 prospects that try our SaaS offering, 1 decides to buy, giving us a 5% conversion ratio. Our model would look as follows:

(bigger)

Since our conversion ratio is 5%, the increase in cost for acquiring a single customer is $1,752.00 in the Windows case! Where does this number come from? Well, there were 19 trials that did not convert to paying customers. Running a 1 month free trial for those prospects cost $87.60 a piece!. The revenue generated by the single converted customer must account for the money spent on the 19 lost prospects. To add even more perspective, if we are generating $250.00 in monthly revenue from that customer, you’d have to keep that customer on board for 7 months in order to recover the money spent on CAC. In fact, that’s being kind and saying that every dollar of revenue can be spent on recovering CAC. This is incorrect, because in reality, you’d have to keep spending $87.60 per month for that EC2 instance for your new paying customer, which needs to be taken off the top of the $250.00, with gross profit applied to CAC recovery. This would stretch out to over a year, and we haven’t even accounted for the remaining customer acquisition cost (you know, the sales and marketing dollars you spent attracting new leads and closing deals). This is not feasible! Sure, you could shorten the trial, maybe ask the customer to foot an evaluation bill, or boost conversion by double, or all of these. At the end of the day, those are all extra hurdles  you could do without, and would be a band-aid that would simply reduce the pain a tiny bit.

Even on the LAMP stack, the results are terrible. You save a couple of months on CAC recovery; big deal. The fundamental problem is that this sort of “architecture” is not built for a more sophisticated delivery model, and “throwing hardware at it” is costly. Lacking multi-tenancy could really cripple your business architecture. This is just the tip of the iceberg. Things that should be simple, like effectively rolling out upgrades, managing orphaned trials, being able to build a revenue stream from data aggregation, are all hampered. We could probably find lots of ways that an ASP approach breaks down your ability to run a business or tune your sales and marketing, but this simple example seems to work as a good illustration of the types of problems to expect.

This topic is close to my heart because of SaaSGrid. As most of you know, SaaSBlogs isn’t really a place where I try to plug our platform, but it’s just so interesting to think that our customers don’t have to worry about multi-tenancy as they can inherit a single instance, multi-tenant architecture from SaaSGrid and avoid these problems without having to spend an additional 6 or 7 figures and 10 or more months in additional project time to get this sort of architecture like some of the older SaaS companies had to achieve good results like this or being able to show people how SaaSGrid can rollout an upgrade across 40,000 end users and a few thousand customers in a matter of 8 minutes because of the multi-tenancy it can inject into otherwise plain web app architectures.

As a SaaS provider, have you experienced similar results painted in the thought experiment? Have you considered issues like this prior to architecture planning? I’d love to hear your thoughts!

If you’d like to mingle with others in the SaaS space, the SaaSBlogs group on LinkedIn now has 3500+ members and is growing every day; make sure you’ re not missing out and join today!

read more

Do We Need Cloud API Standards?

Jun 26, 2010 by Sinclair Schuller

I was on a panel entitled “HYBRID CLOUDS — THE BEST OF BOTH WORLDS?” at the Structure 2010 conference this past week. (Check out a recording of the panel) For those readers who are unfamiliar with the concept of the “hybrid” cloud, essentially, it’s the idea that one logical infrastructure cloud can be composed from two or more clouds of different deployment locale (on-premises vs. elsewhere) or visibility levels (public vs. private)

I was lucky enough to be on this panel with a number of very intelligent people who had some amazing insight into the topic. One of the questions posed to the panel really struck a chord. Essentially, the question was something along the lines of “Are standards necessary for the hybrid cloud to become a reality?” (Roughly minute 29:07 in the recording) The standards being referred to are standards that would abstract the command and control of virtual infrastructure. Each cloud provider (e.g., Amazon through EC2, GoGrid, etc.) typically provides an API by which one can control the deployment and ongoing operations of a virtual resource. Since each cloud provider has a different API, there is a form of lock-in that rears its head since, although each cloud might provide a similar container (say, a Windows 2008 Server), your code for managing and leveraging that cloud is non-portable.

Mårten Mickos, CEO of Eucalyptus, fielded the question stating that standards are necessary for hybrid cloud to really take off and that a standard would accelerate adoption significantly. I agree with the spirit behind Mårten’s view, but not with the idea of formal standards (and as you’ll read, I think Eucalyptus is poised to do some amazing things). Now when we discuss “standards” we mean an industry standard sponsored by a standards body or some sort of consortium. Others on the panel respectfully disagreed with Mårten’s position. Joe Weinman responded that standards aren’t necessary, and that different providers need to be free to evolve solutions that best fit the problem domain and customer need. My position was somewhat in alignment with Joe’s; standards, by definition, must cater to lowest common denominator subsystems which can bias efforts towards standardization rather than on new techniques to solve new problems in the cloud. It’s important to understand why this is the case, and also why it doesn’t mean that although a formal standard shouldn’t be pursued in the near term, generalization should absolutely be pursued.

In some instances, “formal” standards can stifle innovation or distract focus from solving specific problems by becoming abstract enough to be standards compliant, but not specific enough to solve a problem as well as possible. For example, if some cloud provider comes up with a novel way to address something new, they can still conform to the standard, but any uniqueness they bring to the table will be lost behind the abstraction. Just think of the SQL-92 standard. Most every DB vendor has some level of conformity, but various RDBMS’ do something amazingly useful things outside of the SQL-92 standard. If you leverage those proprietary pieces, you can no longer benefit (at least not by much) from the standard itself since you’ve broken conformity. Furthermore, any standards that have evolved in the past have typically fragmented into many sub-standards, dialects, etc. (just like the SQL case)

Now, does this mean that we shouldn’t pursue standardization? Well, if we’re talking about a “formal” standard, we probably don’t need to rush it. Technological generalization, however, is a different story. We’re at a point in the cloud’s evolution that requires organizations like Eucalyptus to focus on providing a coherent means to manage multiple specific cloud implementations. This is amazingly powerful and valuable and doesn’t require a formal standard. Eucalyptus has focused generalizing the Amazon EC2 API as a standard API that could be applied against any other cloud system. Overall, EC2 is a wise choice since it has the broadest market penetration and the most generally applicable API when looking from an abstraction point of view. Eucalyptus is doing some amazing work to make sure that you can leverage API expectations to work with a variety of virtualization clouds. Their efforts will undoubtedly boost adoptability, reduce adoption risk, and increase the viability of private and hybrid cloud deployments. Given Amazon’s API penetration, a market “gold standard” has been created just by shear adoption, and I think people will naturally flock to a proper abstraction like Eucalyptus because of sheer necessity to ensure compatibility with the gold standard while preserving the ability to experiment with new solutions. By choosing EC2, Eucalyptus’ approach provides a standards-like advantage without the baggage since the standard is chasing the market leader, rather than creating a standard and trying to enforce it. Kudos to Mårten and Eucalyptus for helping catalyze adoption of the cloud!

Do you feel that formal API standardization will be absolutely required in order for cloud, private cloud, or hybrid cloud to work, or will things organically tend to a relatively standard oligopoly of well known abstractions without a formal standard in place? Do standards create stifle and slow down innovation by always focusing on the lowest common denominator, or does it promote innovation by guaranteeing a baseline?

read more

What if Salesforce.com weren’t multi-tenant?

May 5, 2009 by Sinclair Schuller

In my last post, I mentioned that I was going to write one last follow up post on the blogosphere-wide multi-tenancy discussion, so here it is! Recently, Salesforce.com quantified some of its architecture, giving us visibility into what it takes to support a SaaS business of its magnitude: 1,000 servers for 55,000 customers and 1.5 million individual subscribers! That’s not too shabby. It also gives us some insight into architecture efficiency. Clearly, there are lots of questions that would need answering to give an accurate analysis, but enough data is there to have good discussion driven by a healthy dose of assumption.

Recently, Bob Warfield over at Smoothspan wrote a couple of excellent multi-tenancy oriented posts. In one of those posts, Warfield drilled into the Salesforce.com numbers I mentioned earlier in an attempt to quantify the impact of their architecture on cost of service. Warfield gave some analysis around multi-tenancy on the overall cost of service incurred by salesforce.com. Looking at the salesforce.com annual report, it’s apparent that it costs salesforce.com $0.12-$0.13 per $1.00 of revenue to deliver its service for a total of $127 million for the year on roughly $1 billion in subscription revenue. Warfield identified that if they’re running 1,000 servers, we can use Amazon pricing as a roughshod estimate of what the actual servers would cost them if they were running on the Amazon cloud: about $4.7 million per year. That leaves about $122 million unaccounted for! As Warfield pointed out, clearly there are lots of other expenses. It’s most likely that these expenses are due to lack of automation and overabundance of personnel due to said lack of automation. To a degree, this means that multi-tenancy is doing its job: the software’s architecture and resource utilization is but a fraction of total cost of service.

To understand this better, let’s use Warfield’s numbers and recast the scenario with the question “What if Salesforce.com weren’t multi-tenant?” Currently, we know that about 55 customers are supported per server. Many propose virtualization as a route to multi-tenancy since it’s so easy to fire up an instance on say, EC2, and create a 1-1 affinity for a customer to a server. If we imagine Salesforce.com to use this architecture instead, one where virtualization is used instead of a multi-tenant approach, we get a drastically different picture. I won’t calculate the EC2 costs, but instead will use Warfield’s numbers: one EC2 instance costs about $4,700 per year ($4.7 million/1,000 instances). If we dedicated a single instance to each customer (that’s 55,000 instances based on Salesforce.com’s number of customers), it would cost Salesforce.com $258 million! Let’s be generous and give it a 2:1 leverage (maybe there is some server sharing) and cut the instances in half, we’re still pushing $129 million. Assuming that this virtualized approach got rid of all internal staff and expenses at Salesforce.com related to cost of service, we’d have to hit that 2:1 leverage to even be competitive with their current cost structure. More than likely, taking a virtualized approach wouldn’t entirely get rid of Salesforce.com’s “out of architecture” expenses since I’m pretty confident that not ALL of the remaining $122 million is dedicated to simply running their 1,000 servers, but some big chunk is. Even if a virtualized or in the cloud approach had a 2:1 leverage and slashed 75%  of the $122 million, Salesforce.com still wouldn’t have the cost profile they do now.

Point? It seems multi-tenancy at $4.7 million worth of hardware/software resources has a big something to do with the overall cost profile. As Warfield pointed out, something else is making up a brunt of the cost, but the architecture approach certainly seems to have depressed it’s part of the cost to something reasonable. Now, how about a multi-tenant architecture on EC2? I’d love to see some cost specifics in the wild on this sort of arrangement!

If you’d like to mingle with others in the SaaS space, the SaaSBlogs group on LinkedIn now has 1950+ members and is growing every day; make sure you’ re not missing out and join today!

read more

What’s More Cost Effective, Piecing a SaaS Stack Together or Using a PaaS?

Mar 19, 2009 by Sinclair Schuller

At the OpSource Summit last week, Treb Ryan gave a great presentation that started with a simple (oft taken for granted) question: what is ‘the cloud’? Although this part of the talk was interesting, one particular part of the talk caught my attention. A third or so of the way through, Treb focused on the drivers behind cloud adoption and discussed how cost reduction is not a primary adoption driver because it doesn’t exist. As evidence, Treb compared two scenarios both requiring a server and equal bandwidth needs. The audience was presented with two implementations: running an EC2 instance to satisfy the server needs and getting a commodity managed server through ServerBeach. Interestingly (but not surprisingly), it was significantly cheaper to go with a raw metal ServerBeach server than an EC2 instance. As Treb went on to point out, the cloud delivers new value that justifies some of that extra cost, like being able to dial up/dial down server instances via an API as needed. This is the ultimate in flexibility, and solves lots of problems. Unfortunately, some are using this flexibility as a hammer and everything around them has become a nail.

During Treb’s talk, I recalled speaking to someone a few weeks earlier about how virtualization solves multi-tenancy (which it does, but very poorly, as I’ll go on to describe) That prompted me to ask: given Treb’s discussion of the higher cost of certain cloud services, what’s the cost of service impact of using something like EC2 as a foundation for a SaaS offering and as the “solution” to the tenancy problem? In fact, what’s the cost of service impact for building a SaaS offering around disparate off the shelf components in general (e.g. Zuora/Aria for billing, something else to help with provisioning + all necessary plumbing code)? I decided to capture this modeling experiment in this post and see if we can come up with a framework for cost of service and capital requirements.

Given that this is a theoretical analysis, we need a scenario to work from. SaaS is broad, so let’s zoom in on a SaaS model we’re all familiar with: charging a per user/per month fee for some line of business application (ala Salesforce.com, RightNow, Basecamp, etc.) A B2B SaaS offering in this model averages around   15 seats per customer. Let’s peg the price per seat at $60.00 per month for the sake of modeling (which by my measure seems reasonable). For our first step of the model, let’s look at recurring cost (cost per unit revenue measure, or in human, cost per seat license) Once we understand this, let’s estimate the total upfront investment for tying all of this together.

To start, let’s focus on EC2. As I mentioned, some like to look to EC2 to solve multi-tenancy. Usually, this is to avoid building a multi-tenant architecture like Salesforce.com’s (which, to do properly & from scratch is very difficult and time consuming) The basic idea is that an EC2 server instance can be fired up per customer (a single instance, single tenant model) or that an application can be split into a couple of layers where each customer might get their own frontend servers for application segregation, but multiple customers map to a single database server each with their own database instance (a hybrid of single instance, single tenant on the frontend and single server, multi tenant database that uses logical databases for segregation). For this hybrid case, let’s assume that we house 40 customers on 1 DB server. In other cases, some might opt for a two server setup: a frontend server and database server per customer, depending on the organization of compute and query processing needs.

So what’s it cost per seat in each of these scenarios, assuming our average? Well, we can do some mapping. First, running a single EC2 instance of the “small server” configuration is $73.00 a month. This might be ok for some, but any other configuration is at least *double* this. We’ll stick to the basics though. At 15 seats per customer, each configuration incurs the following cost penalty per user per month just for EC2:

Per User Server Cost	Cost	as % of Revenue
Single Server	$4.87	8.12%
Two Server	$9.73	16.22%
40-1 Setup	$4.99	8.32%

Ok, so we know what we’re paying for hosting and “multi-tenancy” per user. We’ll reserve judgment on whether this is expensive or not for later.

Now time for billing. Practically every revenue generating SaaS offering needs to issue invoices, collect money, process refunds, etc. Some SaaS providers have cropped up that offer billing as a service. As part of the model, let’s use Zuora. Based on what I found online, Zuora charges 2% of revenue. So, regardless of tenancy scenario, that’s a $1.20 per user charge.

Ok, so we have billing taken care of, but we need to address provisioning of customers as well as a management layer for customer servers. For that, we can tap into Rightscale. Rightscale has technology that lets us capture server images, button click deploy EC2 instances, and seems to have a nice API. Technically, we could build a provisioning system around it that allows us to commission EC2 resources whenever a new customer is onboarded. Rightscale charges $500 per month for up to 20 server instances, plus $24.09 per month per server image managed thereafter. If we add billing and provisioning into the mix, we arrive at the following cost structure for different seat counts:

Customers	5	40	500	2,000
Users	75	600	7,500	30,000
Cost per User
Single Server	$      12.73	$            7.70	$            7.68	$               7.67
Two Server	$      17.60	$          14.18	$          14.15	$            14.15
40-1 Setup	$      12.86	$            7.82	$            7.80	$               7.79

 

So at 30,000 end users with the most efficient setups, we’re hovering around 13% cost of revenue, and this model hasn’t even accounted for other critical pieces. Ouch! This is a huge chunk of revenue to only solve 3 problems. A full SaaS stack is much deeper than this. You’d still have lots of code to write to orchestrate these pieces, and as a delivery stack, it’s quite immature. You still have high availability to deal with, and rolling out updates (which usual requires a good engine to update and migrate customers), user and authentication systems, application health monitoring, etc. All of this stuff piles up into one massive effort, and you then have to maintain a full delivery stack and all the orchestration from and R&D perspective. The R&D requirements are large, and the cost of service continues to grow. This really isn’t optimal, with a huge chunk of service focused on multi-tenancy and hosting via EC2. Generally, efficient multi-tenancy can take a giant bite out of that cost, so using the hammer on the screw might not be the way to go.

This topic interests me because I think it really starts to shine light on why PaaS offerings like SaaSGrid make sense. PaaS offerings capture the most complex pieces of SaaS and trivialize them along with all other critical stack components, while delivering them efficiently so that cost of service goes way down, while rapidly aggregating across apps to boost economies. You could go through the aforementioned design headaches, incur the upfront R&D costs, maintenance costs and relatively high cost of service or you can drop your app into a PaaS container that can do its magic on the app. PaaS costs, particularly over the long run, are insignificant compared to the costs we discussed in this post. I find that folks that “home brew” SaaS delivery capabilities end up spending disproportionate amounts of ongoing R&D on SaaS, on not on generating value for their customers. Personally (and obviously), I like the PaaS idea; after all, we all didn’t go off and build application servers, we relied on things like IIS, Websphere, and JBoss. All of this is important to think about when trying to figure out how to implement your SaaS offering and what each implementation decision means to the bottom line. Granted, this is a theoretical model, but it works well as food for thought.

Are there any other significant components that should be factored into cost of service? Does understanding this make PaaS more appealing? 

The SaaSBlogs group on LinkedIn now has 1660+ members and it’s growing every day; make sure you are not missing out and join today.

read more