
Migrating to Optimizely X Experiments from Optimizely Classic
Split-testing software vendors have traditionally relied on their managed service to win and retain clients.
From Maxymiser to Adobe, Monetate to Qubit, the managed service has been essential to their growth. Even today, most companies cite a lack of resource as the biggest barrier in their optimization program – and a managed service can help overcome that.
Except most managed services suck.
For software vendors, a managed service can throttle their growth and limit their potential. And for their customers, a managed service can lead to substandard results in their conversion optimization program.
And as the optimization and testing industry continues to expand exponentially, this is only going to get worse.
The core of the problem is simple:
Scale is crucial to the success of software vendors. After all, most testing platforms have taken significant investment: Qubit has taken $75M, Monetate $46M, and Maxymiser was acquired by Oracle in August 2015.
But it’s challenging when these companies offer essentially two products – software and service – that scale at very different rates.
With limited cost of sales, a fast-growth software vendor may expect to increase its sales 3–5x in a year.
Look at the rise of Optimizely. Their product’s ease-of-use and their partner program allowed them to focus on the software, not a managed service. And that meant they could grow their market share rapidly:
Between 2012 and 2015, they’ve grown 8x.
Now compare that growth to a marketing services agency. Even a fast-growth mid-size agency may only grow 50% a year – or to put it another way, 1.5x.
If you combine software and service in one company, you’re creating a business that is growing at two very different rates. And this creates a challenge for testing platforms who offer a managed service.
They have three options:
Most will choose option 2 or 3, rather than going all-out on 1. And this choice threatens the quality of their managed service and their ability to scale through partners.
To enable scaling – and to minimise costs – software vendors have to exploit efficiencies at the expense of quality:
In short, to keep up with the growth of the platform, they inevitably have to sacrifice the quality of the managed service in the interest of making it scale.
Let’s look at each of these three points in turn.
At its core, conversion optimization is simple:
Find out why people aren’t converting, then fix it.
The problem is that the first part – finding out why they aren’t converting – is actually pretty hard.
Earlier this year, I shared our take on Maslow’s hierarchy of needs – our “hierarchy of testing”:
The principle is the same as Maslow’s – the layers at the bottom of the pyramid are fundamental.
Starting at the top, there’s no point testing without a strategy. You can’t have a strategy without insight and data to support it. And you can’t get that without defining the goals and KPIs for the project.
In other words, you start at the bottom and work your way up. You don’t jump straight in with testing and hope to get good results.
In particular, the layers in the middle – data and insight – are essential for success. They link the testing program’s goals to the tests. Without them, you’re just guessing.
But all of this comes at a cost – and it’s typically the first cost that managed services cut. Instead of using a similar model to the pyramid above, they jump straight to the top and start testing, without the data and insight to show where and what they should be testing.
Ask them where they get their ideas from, and they’ll probably say heuristics – a nicer way of saying “best practice”.
Creating tests that aren’t based on data and insight is just the start.
To maximise efficiency (again, at the expense of quality), managed services will typically use similar tests across multiple clients. After all, why build a unique test for one client when you can roll it out across 10 websites with only minimal changes?
Break down the fees that managed services charge, and it’s easy to see why they have to do this.
Let’s assume Vendor X is charging £3k to deliver 2 tests per month. If we allow £1k/day as a standard managed service rate, then that gives 24 hours – or 12 hours per test.
At Conversion.com, we know that even just to build an effective test can take longer than 12 hours – and that’s before you add in time for strategy, design, QA and project management.
The cookie-cutter approach is problematic for two core reasons:
This focus on efficiencies means that account managers are able to work across at least 10–20 clients. Even assuming that account managers are working at 80% utilisation, that means that clients are getting between 1.5 and 3 hours of their time each week.
Is that a problem?
At Conversion.com, our consultants manage 3–5 clients in total. We feel that limit is essential to deliver an effective strategy for optimization.
Ultimately, it reflects our belief that conversion optimization can and should be integral to how a company operates and markets itself – and that takes time.
Conversion optimization should let you answer questions about your commercial, product and marketing strategy:
Not “Which colour button might work best?”
Conversion optimization isn’t a series of tactical cookie-cutter tests that can be churned out for your website, while 19 other clients compete for your AM’s attention.
It’s not surprising that a managed service with a “one-size-fits-most” approach for its clients doesn’t perform as well as testing strategy from a dedicated optimization agency.
The difference in approach is reflected in results (and, of course, the cost of the service).
But some managed services are misleading their clients over the success of their testing program.
There are three warning signs that the value of a managed service is being overreported:
Managed service has a place. It can be essential to expand a new market – especially where the product’s learning curve is steep and may limit its appeal to a self-serve audience.
But the focus should always be on the quality of the service. Vendors can subsidise the cost of their service if needed – whether through funding or the higher profit margin in software – to deliver an effective optimization program.
Then, their growth should come through self-service and partners. As above, service and software scale at different rates – and the faster a software vendor champions self-service and a partner program, the faster they’ll grow.
Disclaimer: I’m the CEO of Conversion.com, an agency that specialises in conversion optimization. We partner with many of the software vendors above. While we have a vested interest in companies choosing us over managed service, we have an even greater interest in making sure they’re testing effectively.