From quick wins to cultural shifts - understanding the experimentation maturity model

There are many ways you could attempt to measure conversion optimization and experimentation maturity. At Conversion.com we work with businesses and teams at all levels of conversion maturity – from businesses just starting out with conversion optimization that have never launched a test, to businesses with growth and optimization teams of hundreds of people. From this experience we’ve built up a good picture of what defines maturity.

Our model for maturity focuses on measuring strategic maturity. We believe conversion optimization maturity shouldn’t be limited by the size of your organization, team or budget. Any organization, armed with an understanding of what maturity looks like, where they are currently and what level they would like to reach, should be able to reach the higher stages of experimentation maturity.

For this reason our model does not include basic measures of scale such as number of tests launched per month or size of experimentation team. Nor does it refer to any specific tools or pieces of technology as requirements. In defining this model we wanted to keep things simple. To create a model for maturity that helps to start conversations both with our clients and in any team serious about putting experimentation at the heart of their business.

Our model measures maturity against three key scales: experimentation goals, experimentation strategy and data and technology.

What are the goals of your optimization program?
If you’re just starting to explore experimentation and conversion optimization you might have the goal for your program of simply getting a test live. At the other end of the spectrum, more businesses are emerging now where the goal of experimentation is to be a driving force in the overall strategy of the business. The goals that we set for experimentation in our organizations,and our ambition in this area set the tone for how we approach and deliver experimentation. Organisations that have embraced experimentation set more ambitious goals and these goals require a more mature approach to achieve them. That’s why evaluating the goals for experimentation within your own organization is the best place to start when evaluating your place on the maturity scale.

Developing your maturity in this area involves shifting the scope of your goals and developing alignment of the goals of experimentation with the overall goals of your business. Moving from goals being about short-term results and impact on KPIs, towards being about answering business questions and informing business decisions and strategy.

It’s important to make a distinction between reality and ambition when trying to plot your current position in this scale. Consider the role that experimentation currently plays in your organization and how you are currently setting the goals, rather than what you’d like to be your goal for experimentation in an ideal world. The maturity model is most useful as a tool for assessing where you are now, where you want to be in the future, and what needs to change to close the gap between the two.

Where does your strategy for experimentation come from?
Experimentation goals and experimentation strategy are closely linked, with strategy being how you achieve the goals you’ve set. If you are just starting to explore experimentation, you may not have thought too much yet about an overall strategy. Early on, experimentation strategy tends to be largely tactical in nature, with ideas generated on an ad-hoc basis and experiment prioritization based on most urgent priority or a simple impact/ease model. Each experiment is treated as an individual exercise.

Advanced optimization teams plan their strategy for achieving their optimization goals across both the short-term and long-term. Long-term strategic planning should focus on prioritization at the high level of goals and priorities. Conversion optimization is an ongoing process. It’s not possible to do everything at once, and mature teams plan and prioritize the areas that they will focus on right now and those that they’ll focus on later in the year. In this way they can keep their focus narrow and ensure there is a clear plan for achieving their goals.

Advanced optimization teams view testing not as a tool for increasing conversion rates but as a tool for answering questions. Starting with the big picture, they identify the business questions that need to be answered. They then break these problems down to define the tests and research that they need to complete to validate their hypotheses and answer that question.

As we move up the maturity stages, optimization strategy becomes more thematic. Experiments are considered now as one tool for exploring a specific theme or conversion lever. At this level, experimentation is organised as a series of projects, each made up of a combination of targeted user-research pieces and experiments. These projects align to business strategy, and experimentation starts to play a leading role in overall business strategy.  

How do you detect and measure the things that matter?
The quality of insight gained from experimentation is directly correlated to the quality of data that you collect about what happened. If your goal is just to get some experiments live there is probably less emphasis on ensuring those experiments have a solid grounding in data. Ensuring the data the experiments produce when they do run is reliable and actionable can often be more of an afterthought. Advanced optimization teams will be a lot more deliberate, with data and insight playing leading roles in generating test hypotheses, and experiment data being a valuable source of insight for the business and the people in it. Maturity here is being confident in your data so that you can challenge it, ask probing questions of experiment impact, and be able to confidently produce the answers.

Technology plays a key role in this, but is only as good as the strategy for using it. The specific tools you use aren’t as important, for example, as your ability to connect your tools and data sets together. A set of simple but connected tools can deliver greater quality of insight that one advanced but isolated tool. Start with your experimentation tool, and connect it to any other tools you have such as surveys, session recording and heatmaps. In particular, connect it to your back-end reporting systems so that the impact of experiments can be measured against the KPIs that really matter, and that people look at on a daily basis.

Now that we’ve explored the 3 scales that we use to measure maturity we can define approximate levels of maturity to give us an overall scale and tool for evaluating our own place. Really though, maturity is a continuous scale rather than something discreetly split into levels. When reviewing the levels below you may place yourself at different levels for each of the 3 scales. This is very common. There is often one part of our approach that we know is probably holding us back – a weak link in the chain. This model should help formalise and pinpoint that weakness and start the conversations for how to overcome it.

If you’re looking to develop the maturity of your experimentation and conversion optimization strategy then we’d be happy to help. Just drop an email to hello@conversion.com and we’ll organise a free maturity consultation with one of our team.