Author: Kyle Hearnshaw

Everyone approaches experimentation differently. But there’s one thing companies that are successful at experimentation all have in common: a strategic framework that drives experimentation.

In the last ten years we’ve worked with start-ups through to global brands like Facebook, the Guardian and Domino’s Pizza, and the biggest factor we’ve seen impact success is having this strategic framework to inform every experiment.

In this post, you’ll learn

• • Why a framework is crucial if you want your experimentation to succeed

• • How to set a meaningful goal for your experimentation programme

• How to build a framework around your goal and create your strategy for achieving it

We’ll be sharing the experimentation framework that we use day in, day out with our clients to deliver successful experimentation projects. We’ll also share some blank templates of the framework at the end, so after reading this you’ll be able to have a go at completing your own straight away.

Why use a framework? Going from tactical to strategic experimentation

Using this framework will help you mature your own approach to experimentation, make a bigger impact, get more insight and have more success.

Having a framework:

• • • Establishes a consistent approach to experimentation across an entire organisation, enabling more people to run more experiments and deliver value
  • • Allows you to spend more time on the strategy behind your experiments and less time on the “housekeeping” of trying to manage your experimentation programme.
  • Enables you to transition from testing tactically to testing strategically.

Let’s explore that last point in detail.

In tactical experimentation every experiment is an island – separate and unconnected to any others. Ideas generally take the form of solutions – “we should change this to be like that” and come from heuristics (aka guessing), best practice or from copying a competitor. There is very little guiding what experiments run where, when and why.

Strategic experimentation on the other hand is focused on achieving a defined goal and has clear strategy for achieving it. The goal is the starting point – a problem with potential solutions explored through the testing of defined hypotheses. All experiments are connected and experimentation is iterative. Every completed experiment generates more insight that prompts further experiments as you build towards achieving the goal.

If strategic experimentation doesn’t already sound better to you then we should also mention the typical benefits you’ll see as a result of maturing your approach in this way.  

• • You’ll increase your win rate – the % of experiments that are successful

• • You’ll increase the impact of each successful experiment – on top of any conversion rate uplifts, experiments will generate more actionable insight

• You’ll never run out of ideas again – every conclusive experiment will spawn multiple new ideas

Introducing the Conversion.com experimentation framework

As we introduce our framework, you might be surprised by its simplicity. But all good frameworks are simple. There’s no secret sauce here. Just a logical, strategic approach to experimentation.

Just before we get into the detail of our framework a quick note on the role of data. Everything we do should be backed by data. User-research and analytics are crucial sources of insight used to build the layers in our framework. But the experiments we run using the framework are often the best source of data and insight we have. An effective framework should therefore minimise the time it takes to start experimenting. We cannot wait for perfect data to appear before we start, or try and get things right first time. The audiences, areas and levers that we’ll define in our framework come from our best assessment of all the data we have at a given time. They are not static or fixed. Every experiment we run helps us improve and refine them and our framework and strategy is updated continuously as more data becomes available.

Part 1 – Establishing the goal of your experimentation project

The first part of the framework is the most important by far. If you only have time to do one thing after reading this post it should be revisiting the goal of your experimentation.

Most teams don’t set a clear goal for experimentation. It’s a simple as that. Any strategy needs to start with a goal, otherwise how can you differentiate success from wasted effort?

A simple test of whether your experimentation has a clear goal is to ask everyone in your team to explain it. Can they all give exactly the same answer? If not, you probably need to work on this. 

Don’t be lazy and choose a goal like “increase sales” or “growth”. We’re all familiar with the importance of goals being “SMART” (specific, measurable, achievable, relevant, time-bound) when setting personal goals. Apply this when setting the goal for experimentation.

Add focus to your goal with targets, measures and deadlines, and wherever possible be specific rather than general. Does “growth” mean “increase profit” or “increase revenue”? By how much? By when? A stronger goal for experimentation would be something like “Add an additional £10m in profit within the next 12 months”. There will be no ambiguity as to whether you have achieved that or not in 12 months’ time.

Some other examples of strong goals for experimentation

• • “Increase the rate of customers buying add-ons from 10% to 15% in 6 months.”

• • “Find a plans and pricing model that can deliver 5% more new customer revenue before Q3”

• “Determine the best price point for [new product] before it launches in June.”

A clear goal ensures everyone knows what they’re working towards, and what other teams are working towards. This means you can coordinate work across multiple teams and spot any conflicts early on.

Part 2 – Defining the KPIs that you’ll use to measure success

When you’ve defined the goal, the next step is to decide how you’re going to measure it. We like to use a KPI tree here – working backwards from the goal to identify all the metrics that affect it.

For example, if our goal is “Add an additional £10m in profit within the next 12 months” we construct the KPI tree of the metrics that combine to calculate profit. In this simple example let’s say profit is determined by our profit per order times how many orders we get, minus the cost of processing any returns.

These 3 metrics then break down into smaller metrics and so on. You can then decide which of the metrics in the tree you can most influence through experimentation. These then become your KPIs for experimentation. In our example we’ve chosen average order value, order conversion rate and returns rate as these can be directly impacted in experiments. Cost per return on the other hand might be more outside our control.

When you’re choosing KPIs, remember what the K stands for. These are key performance indicators – the ones that matter most. We’d recommend choosing at most 2 or 3. Remember, the more you choose, the more fragmented your experimentation will be. You can track more granular metrics in each experiment, but the overall impact of your experiments will need to be measured in these KPIs.

Putting that all together, you have the first parts of your new framework. This is our starting point – and it is worth the time to get this right as everything else hinges on this.

Part 3 – Understanding how your audience impacts your KPIs and goal

Now we can start to develop our strategy for impacting the KPIs and achieving the goal. The first step is to explore how the make-up of our audience should influence our approach.

In any experiment, we are looking to influence behaviour. This is extremely difficult to do. It’s even more difficult if we don’t know who we’re trying to influence – our audience.

We need to understand the motivations and concerns of our users – and specifically how these impact the goal and KPIs we’re trying to move. If we understand this, then we can then focus our strategy on solving the right problems for the right users.

So how do we go about understanding our audience? For each of our KPIs the first question we should ask is “Which groups of users have the biggest influence on this KPI?” With this question in mind we can start to map out our audience.

Start by defining the most relevant dimensions – the attributes that identify certain groups of users. Device and Location are both dimensions, but these may not be the most insightful ways to split your audience for your specific goal and KPIs. If our goal is to “reduce returns by 10% in 6 months”, we might find that there isn’t much difference in returns rate for desktop users compared to mobile users. Instead we might find returns rate varies most dramatically when we split users by the Product Type that they buy.

For each dimension we can then define the smaller segments – the way users should be grouped under that dimension. For example, Desktop, Mobile and Tablet would be segments within the Device dimension.

You can have a good first attempt at this exercise in 5–10 minutes. At the start, accuracy isn’t your main concern. You want to generate an initial map that you can then start validating using data – refining your map as necessary. You might also find it useful to create 3 or 4 different audience maps, each splitting your audience in different ways, that are all potentially valid and insightful for your goal.

Once you have your potential audiences the next step would then be to use data to validate the size and value of these audiences. The aim here isn’t to limit our experiments to a specific audience – we’re not looking to do personalisation quite yet. But understanding our audiences means when we come to designing experiments we’ll know how to cater to the objections and concerns of as many users as possible.

Part 4 – Identifying the areas with the greatest opportunity to make an impact

Armed with an better understanding of our audience, we still need to choose when and where to act to be most effective. Areas is about understanding the user journey – and focusing our attention on where we can make the biggest impact.

For each audience, the best time and place to try and influence users will vary. And even within a single audience, the best way to influence user behaviour is going to depend on which stage of their purchase journey the users are at.

As with audiences, we need to map out the important areas. We start by mapping the onsite journeys and funnels. But we don’t limit ourselves to just onsite experience – we need to consider the whole user journey, especially if our goal is something influenced by behaviours that happen offsite. We then need to identify which steps directly impact each of our KPIs. This helps to limit our focus, but also highlights non-obvious areas where there could be value.

As with audiences, you can sketch out the initial map fairly quickly, then use analytics data to start adding more useful insights. Label conversion and drop-off rates to see where abandonment is high. Don’t just do this once for all traffic, do this repeatedly, once for each of the important audiences identified in the previous step. This will highlight where things are similar but crucially where things are different.

So with a good understanding of our audiences and areas we can add these to our framework. Completing these two parts of the framework is easier the more data you have. Start with your best guess at the key audiences and areas, then go out and do your user-research to inform your decisions here. Validate your audiences and areas with quant and qual data.

Part 5 – Identifying the potential levers that influence user behaviour

Levers are the factors we believe can influence user behaviour: the broad themes that we’ll explore in experimentation. At its simplest, they’re the reasons why people convert, and also the reasons why people don’t convert. For example, trust, pricing, urgency and understanding are all common levers.

To identify levers, first we look for any problems that are stopping users from converting on our KPI – we call these barriers to conversion. Some typical barriers are lack of trust, price, missing information and usability problems.

We then look for any factors that positively influence a user’s chances of converting – what we call conversion motivations. Some typical motivations are social proof (reviews), guarantees, USPs of the product/service and savings and discounts.

Together the barriers and motivations give us a set of potential levers that we can “pull” in and experiment to try and influence behaviour. Typically we’ll try to solve a barrier or make a motivation more prominent and compelling.

Your exact levers will be unique to your business. However there are some levers that come up very frequently across different industries that can make for good starting points.

Ecommerce – Price, social proof (reviews), size and fit, returns, delivery cost, delivery methods, product findability, payment methods, checkout usability

Saas – Free trial, understanding product features, plan types, pricing, cancelling at the end of trial, monthly vs annual pricing, user onboarding

Gaming – welcome bonuses, ongoing bonuses, payment methods, popular games, odds

Where do levers come from? Data. We conduct user-research and gather quantitative and qualitative data to look for evidence of levers. You can read more about how we do that here.

When first building our framework it’s important to remember that we’re looking for evidence of levers, not conclusive proof. We want to assemble a set of candidate levers that we believe are worth exploring. Our experiments will then validate the levers and give us the “proof” that a specific lever can effectively be used to influence user behaviour.

You might start initially with a large set of potential levers – 8 or 10 even. We need a way to validate levers quickly and reduce this set down to the 3–4 most effective. Luckily we have the perfect tool for that in experiments.

Part 6 – Defining the experiments to test your hypotheses

The final step in our framework is where we define our experiments. This isn’t an exercise we do just once – we don’t define every experiment we could possibly run from the framework at the start – but using our framework we can start to build the hypotheses that our experiments will explore.

At this point, it’s important to make a distinction between a hypothesis for an experiment and the execution of an experiment. A hypothesis is a statement we are looking to prove true or false. A single hypothesis can then be tested through the execution of an experiment – normally a set of defined changes to certain areas for an audience.

We define our hypothesis first before thinking about the best execution of an experiment to test it, as there are many different executions that could test a single hypothesis. At the end of the experiment the first thing we do is use the results to evaluate whether our hypothesis has been proven or disproven. Depending on this, we then evaluate the execution separately to decide whether we can iterate on it – to get even stronger results – or whether we need to re-test the hypothesis using a different execution.  

The framework makes it easy to identify the hypothesis statements that we will look to prove or disprove in our experiments. We can build a hypothesis statement from the framework using this simple template

“We believe lever [for audience] [on area] will impact KPI.”

The audience and area here are in square brackets to denote that it’s optional whether we want to specify a single audience and area in our hypothesis. Doing so will give us a much more specific hypothesis to explore, but in a lot of cases we may also be interested in testing the effectiveness of the lever across different audiences and different areas – so may want to not specify the audience an area until we define the execution of the experiment.

Using the framework

Your first draft of the completed framework will have a large number of audiences, areas and levers, and even multiple KPIs. You’re not going to be able to tackle everything at once. A good strategy should have focus. Therefore you need to do two things before you can define a strategy from the framework.

Prioritise KPIs, audiences and areas

We’re going to be publishing a detailed post of how this framework enables an alternative approach to prioritisation than typical experiment prioritisation.

The core idea is that you need to first prioritise the KPI you most need to impact from your framework in order to achieve your goal. Then evaluate your audiences identify those groups that are the highest priority groups to influence if we want to move that KPI. Then for that audience prioritise those areas of the user-journey that offer the greatest opportunity to influence their behaviour.

This then gives you a narrower initial focus. You can return to the other KPIs at a later date and do the same prioritisation exercise for them.

Validate levers

You need to quickly refine your set of levers and identify the ones that have the greatest potential. If you have run experiments before you should look back through each experiment and identify the key lever (or levers) that were tested. You can then give each lever a “win rate” based on how often experiments using that lever have been successful. If you haven’t yet started experimenting, you likely already have an idea of the potential priority order of your levers based on the volume of evidence for each that you found during your user-research.

However, the best way to validate a lever is to run an experiment to test the impact it can have on our KPI. You need a way to do this quickly. You don’t want to invest significant time and effort testing hypotheses around a lever that turns out not have ever been valid. Therefore for each lever you should identify what we call the minimum viable experiment.

You’re probably familiar with the minimum viable product (MVP) concept. In a minimum viable experiment we look to design the simplest experiment we can that will give us a valid signal as to whether a lever works at influencing user behaviour.

If the results of the minimum viable experiment show a positive signal, we can then justify investing further resource on more experiments to validate hypotheses around this lever. If the minimum viable experiment doesn’t give a positive signal, we might then de-prioritise that lever, or remove it completely from our framework. We’ll also be sharing a post soon going into detail on designing minimum viable experiments.

Creating a strategy

How you create a strategy from the framework will depend on how much experimentation you have done before and therefore how confident you are in your levers. If you’re confident in your levers then we’d recommend defining a strategy that lasts for around 3 months and focuses on exploring the impact of 2-3 of your levers on your highest priority KPI. If you’re not confident in your levers, perhaps having not tested them before, then we’d recommend an initial 3-6 month strategy that looks to run the minimum viable experiment on as many levers as possible. This will enable you to validate your levers quickly so that you can take a more narrow strategy later.

Crucially at the end of each strategic period we can return to the overall framework, update and refine it from what we’ve learnt from our experiments, and then define our strategy for the next period.

Key takeaways

You can have a first go at creating your framework in about 30 minutes. Then you can spend as long or as little time as you like refining it before you start experimenting. Remember your framework is a living thing that will change and adapt over time as you learn more and get more insight.

1. Establish the goal of your experimentation project
2. Define the KPIs that you’ll use to measure success
3. Understand how your audience impacts your KPIs and goal
4. Identify the areas with the greatest opportunity to make an impact
5. Identify the potential levers that influence user behaviour
6. Define the experiments to test your hypotheses

The most valuable benefit of the framework is that it connects all your experimentation together into a single strategic approach. Experiments are no longer islands, run separately and with little impact on the bigger picture. Using the framework to define your strategy ensures that every experiment is playing a role, no matter how small, in helping you impact those KPIs and achieve your goal.

Alongside this, using a framework also brings a large number of other practical advantages:

• It’s clear– your one diagram can explain any aspect of your experimentation strategy to anyone that asks or if you need to report on what you’re doing
• It acts as a sense check – any experiment idea that gets put forward can be assessed based on how it fits within the framework. If it doesn’t fit, it’s easy rejection with a clear reason why
• It’s easy to come back to – things have a nasty habit of getting in the way of experimentation, but with the framework even if you leave it for a couple of months, it’s easy to come back to it and pick up where you left off
• It’s easier to show progress and insight – one of the biggest things teams struggle with is documenting the results of all their experiments and what was learnt. With the framework, the idea is that the framework updates and changes over time so you know that your previous experiment results have all been factored in and you’re doing what you’re doing for a reason

As we said at the start of this post, there is no special sauce in this framework. It’s just taking a logical approach, breaking down the key parts of an experimentation strategy. The framework we use is the result of over 10 years of experience running experimentation and CRO projects and it looks how it does because it’s what works for us. There’s nothing stopping you from creating your own framework from scratch, or taking ours and adapting it to suit your business or how your teams work. The important thing is to have one, and to use it to go from tactical to strategic experimentation.

You can find a blank Google Slide of our framework here that you can use to create your own.

Alternatively you can download printable versions of the framework if you prefer to work on paper. These templates also allow for a lot more audiences, areas, levers and experiments than we can fit in a slide.

If you would like to learn more, get in touch today!

At Conversion.com, our team and our clients know first-hand the impact experimentation can have. But we also see all too often the simple mistakes, misconceptions and misinterpretations organisations make that limit the impact, effectiveness and adoption of experimentation.

We wanted to put that right. But we didn’t just want to make another best-practice guide to getting started with CRO or top 10 tips for better experiments. Instead, inspired by the simple elegance of the UK government design principles, we set ourselves the challenge of defining a set of the core experimentation principles.

Our ambition was to create a set of principles that, if followed, should enable anyone to establish experimentation as a problem solving framework for tackling any and all problems their organisation faces. To distill over 10 years of experience in conversion optimisation and experimentation down to a handful of principles that address every common mistake, every common misconception and misinterpretation of what good experimentation looks like.

Many hours of discussion, debate and refinement later, we’re happy to be able to share the end product – the 9 principles of experimentation.

Here are the principles in their simplest form. You can also download a pdf of the experimentation principles that also includes quotes and stories we’ve gathered from experimentation experts at companies such as Just Eat, Booking.com, Microsoft and Facebook. A few snippets of those quotes are included below as a taster.

DOWNLOAD PRINCIPLES PDF

1 – Challenge assumptions, beliefs and doctrine

Experimentation should not be limited to optimising website landing pages, funnels and checkouts. Use experimentation as a tool to challenge the widely held assumptions, ingrained beliefs and doctrine of your organisation. It’s often by challenging these assumptions that you’ll see the biggest returns. Don’t accept “that’s the way it’s always been done” -to do so is to guarantee you’ll get the results you’ve always had. Experimentation provides a level playing field for evaluating competing ideas, scientifically, without the influence of authority or experience.

-Rand Fishkin, CEO and Co-founder, SparkToro

2 – Always start with data

It sounds trite to say you should start with data. Yet most people still don’t. Gut-feel still dominates decision making and experiments based on gut-feel rarely lead to meaningful impact or insight. Good experimentation starts with using data to identify and understand the problem you’re trying to solve. Gather data as evidence and build a case for the likely causes of those problems. Once you have gathered enough evidence you can start to formulate hypotheses to be proven or disproven through experiments.

3 – Experiment early and often

In any project, look for the earliest opportunity to run an experiment. Don’t wait until you have already built the product/feature to run an experiment, or you’ll find yourself moulding the results to justify the investment or decisions you’ve already made. Experiment often to regularly sense-check your thinking, remove reliance on gut-feel and make better informed decisions.

4 – One, provable hypothesis per experiment

Every experiment needs a single hypothesis. That hypothesis statement should be clear, concise and provable – a cause-effect statement. A single hypothesis ensures the experiment results can be used to evaluate that hypothesis directly. Competing hypotheses introduce uncertainty. If you have multiple hypotheses, separate these into distinct experiments.

5 – Define the success metric and criteria in advance

Define the primary success metric and the success criteria for an experiment at the same time that you define the hypothesis. Doing so will focus your exploration of possible solutions around their ability to impact this metric. Failing to do so will also introduce errors and bias when analysing results—making the data fit your own preconceived ideas or hopes for the outcome.

-Lukas Vermeer, Booking.com

6 – Start with the minimum viable experiment, then iterate

When tackling complex ideas the temptation can be to design a complex experiment. Instead, look for the simplest way to run an experiment that can validate just one part of the idea: the minimum viable experiment. Run this experiment to quickly get data or insight that either gives the green light to continue to more complex implementations, or flags problems early on. Then iterate and scale to larger experiments with confidence that you’re heading in the right direction.

7 – Evaluate the data, hypothesis, execution and externalities separately

When faced with a negative result, it can be tempting to declare an idea dead-in-the-water and abandon it completely. Instead, evaluate the four components of the experiment separately to understand the true cause:

1. The data – was it correctly interpreted?
2. The hypothesis – has it actually been proven or disproven?
3. The execution – was our chosen solution the most effective?
4. External factors – has something skewed the data?

An iteration with a slightly different hypothesis, or an alternative execution could end in very different results. Evaluating against these four areas separately, for both negative and positive results, gives four areas on which you can iterate and gain deeper insight.

8 – Measure the value of experimentation in impact and insight

The ultimate judge of the value of an experimentation programme are the impact it delivers and the insight it uncovers. Experimentation can only be judged a failure if it doesn’t give us any new insight that we didn’t have before. Negative results that give us new insight can often be more valuable than positive results that we don’t understand.   

9 – Use statistical significance to minimise risk

Use measures of statistical significance when analysing experiments to manage the risk of making incorrect decisions. Achieving 95% statistical significance leaves a 1 in 20 chance of a false positive – seeing a signal where there is no signal. This might not be acceptable for a very high risk experiment with something like product or pricing strategy, so increase your requirements to suit your appetite. Beware experimenting without statistical significance, that’s not much better than guessing.

The best data scientists are skeptics that double-check, triangulate results, and evaluate the positive and the negative results with the same scientific rigor Click To Tweet

-Ron Kohavi, Microsoft

***

These are the 9 principles we felt most strongly define experimentation, but no doubt we could have added others and made a longer list. If you have experimentation principles that you use at your organisation that we haven’t included here we’d be interested to hear about them and why you feel they’re important.

For more detail and even more insights from some of the world leading experts on experimentation, please be sure to download the full experimentation principles.

**

We’re also looking for more stories and anecdotes of both good and bad examples of these principles in action from contributors outside Conversion to include in our further iterations of these principles. If you have something you feel epitomises one of these principles then please get in touch and you could feature in our future posts and content about these principles.

And finally, if you want to be notified when we publish more content about these experimentation principles, drop us an email with your contact details.

For any of the above get in touch at hello@conversion.com.

DOWNLOAD PRINCIPLES PDF

The ‘year of personalisation’ has been on the cards for a while now. 

A quick Google search, and you’ll find plenty of articles touting 201X as the year that personalisation will take off. Midway through 2017, we’re still waiting for it to really take hold.

So, what’s holding personalisation back from becoming the norm? Why isn’t every website already perfectly tailored to my individual needs?

There are two main reasons we have yet to see personalisation live up to the great expectations.

The first reason is the expectation itself. The dream of personalisation as it’s sold – a website responsive to the user’s habits – will likely remain just that for all but a handful of organisations that meet very challenging criteria.

The rest of us have more realistic and practical expectations for personalisation where it must prove its worth against many other activities competing for resources.

The second reason is that implementing personalisation is a difficult process, and one where it makes sense to start small and build up. No doubt the majority of organisations are starting to explore personalisation, but the reason we feel it has yet to take off is because they are still in the early stages. Personalisation is hard. It’s not something that can be undertaken lightly and from a conversion optimisation perspective, is only possible if you have already reached the higher levels of experimentation maturity.

So, how do I know if my business is ready? 

Before even thinking about technical capabilities, tools or technology, you should evaluate personalisation in three areas: suitability, profitability and maturity.

Certain types of operating model are better suited to personalisation and offer more opportunity and potential. If you maintain an ongoing relationship with your customers and see a high frequency of engagement e.g. if you get a lot of repeat transactions as an e-commerce site, then personalisation is likely to be more suitable. In general, the greater the frequency of your customers’ visits, the more relevant any previous data about that customer is likely to be, and the experience you create for that customer can be more relevant as a consequence.

On the other hand, for websites that focus on a single engagement, where repeated engagement is unlikely or infrequent, personalisation is likely to be far less effective. Those organisations are likely to have limited data about the user and, consequently, it will be more difficult to create highly relevant experiences. Depending on what model your organisation operates, you might decide that your website is more or less suited to personalisation.

Implementing and maintaining personalisation comes with considerable costs. You should only invest in personalisation if you can demonstrate that the benefits will outweigh the costs involved. The underlying hypothesis of personalisation is that delivering a more relevant experience to a user will increase the likelihood of them converting. As with all hypotheses, this should be tested and validated. Experimentation and testing will allow you to prove the value of personalisation for your business, so that is where you should start.

Personalisation requires a deep understanding of user behaviour. More so than in A/B testing, we need to understand not just why users aren’t converting, but also how segments of users vary in their motivation, ability and trigger. If your organisation is still at the lower levels of experimentation and conversion optimisation maturity, then it will be difficult to implement personalisation experimentation in a way that is effective and manageable. A good way to think of it is as a higher level of experimentation maturity that you should explore once you have exhausted the gains that could be had from general experimentation and conversion rate optimisation.

What is a realistic expectation for personalisation for my business?

It doesn’t have to be the 1-to-1 highly granular customisation that people tend to think it is. There are many different ways to approach personalisation and the approach that is best for your business will depend on a number of factors.

In order to start your discussions about personalisation, here are a few different types that you may want to explore:

• Behaviour-based personalisation – This is a great place to start as it has a low barrier to entry. Generally, this type of personalisation is based on the user’s behaviour on the site during their current session. Altering the content that you show the user when they return to the homepage based on what type of pages they have visited in this session, (or multiple sessions using cookies), for example.
• Context-based personalisation – This is where the user’s experience is personalised based on the context of their visit to the site. A basic example of this is personalising landing pages based on the user’s PPC search term, the email they clicked through, or the display ad they’ve clicked. This is more commonly known as segmentation, but really this is just another type of personalisation. This can be a good step towards defining the important audiences/segments that would then feature in more advanced personalisation.
• Attribute-based personalisation – This is what most people think of when they think about personalisation: using prior knowledge or attributes about a user to personalise their experience. This type generally requires more advanced technology to connect sources of data about a user together in a way that creates what’s known as a Dynamic Customer Profile for each user. This profile will contain all the possible attributes around which an experience can be personalised to that specific user.
• User-led personalisation – Not all personalisation has to be invisible to the user. In fact, it could be argued that personalisation is more effective when the user can see it happening and is aware that the site is being customised to them. Netflix users know that movie recommendations are based on what they’ve already watched, just as Amazon’s product recommendations are based on what you’ve previously shown an interest in purchasing. This feels more compelling than if you were just shown recommended products without reason.
• Personalisation via predictive modelling – This is the realm of AI and machine learning, where models can be used to assign a user to the best guess ‘lookalike’ audience based on their first few actions on the site. For example, users that visit the ‘Sale’ section of a site within the first three clicks could be assumed to fit in a ‘bargain hunter’ audience. Then any previous learnings about how to effectively convert bargain hunters could be applied to personalise the experience for this user.

So, will 2018 finally be the ‘year of personalisation’?

I expect we will see a lot more case studies emerging of personalisation proving successful as more organisations start seeing the rewards of their investment in this area. If nothing else, I’d expect 2018 to be the year that organisations individually make their decision whether to invest or not invest in personalisation in a serious way.

Personalisation isn’t going to be suitable for everyone. The dream of 1-to-1 personalisation that runs itself might remain a dream for the majority, but taking the first steps towards investigating its potential is an exercise that every organisation should undertake. As preparation, plotting your current position on our experimentation maturity model will help you to plan the steps you need to take to be ready when the time comes.

There are many ways you could attempt to measure conversion optimisation and experimentation maturity. At Conversion.com we work with businesses and teams at all levels of conversion maturity – from businesses just starting out with conversion optimisation that have never launched a test, to businesses with growth and optimisation teams of hundreds of people. From this experience we’ve built up a good picture of what defines maturity.

Our model for maturity focuses on measuring strategic maturity. We believe conversion optimisation maturity shouldn’t be limited by the size of your organisation, team or budget. Any organisation, armed with an understanding of what maturity looks like, where they are currently and what level they would like to reach, should be able to reach the higher stages of experimentation maturity.

For this reason our model does not include basic measures of scale such as number of tests launched per month or size of experimentation team. Nor does it refer to any specific tools or pieces of technology as requirements. In defining this model we wanted to keep things simple. To create a model for maturity that helps to start conversations both with our clients and in any team serious about putting experimentation at the heart of their business.

Our model measures maturity against three key scales: experimentation goals, experimentation strategy and data and technology.

Experimentation goals

What are the goals of your optimisation programme?
If you’re just starting to explore experimentation and conversion optimisation you might have the goal for your programme of simply getting a test live. At the other end of the spectrum, more businesses are emerging now where the goal of experimentation is to be a driving force in the overall strategy of the business. The goals that we set for experimentation in our organisations,and our ambition in this area set the tone for how we approach and deliver experimentation. Organisations that have embraced experimentation set more ambitious goals and these goals require a more mature approach to achieve them. That’s why evaluating the goals for experimentation within your own organisation is the best place to start when evaluating your place on the maturity scale.

Developing your maturity in this area involves shifting the scope of your goals and developing alignment of the goals of experimentation with the overall goals of your business. Moving from goals being about short-term results and impact on KPIs, towards being about answering business questions and informing business decisions and strategy.

It’s important to make a distinction between reality and ambition when trying to plot your current position in this scale. Consider the role that experimentation currently plays in your organisation and how you are currently setting the goals, rather than what you’d like to be your goal for experimentation in an ideal world. The maturity model is most useful as a tool for assessing where you are now, where you want to be in the future, and what needs to change to close the gap between the two.

Experimentation strategy

Where does your strategy for experimentation come from?
Experimentation goals and experimentation strategy are closely linked, with strategy being how you achieve the goals you’ve set. If you are just starting to explore experimentation, you may not have thought too much yet about an overall strategy. Early on, experimentation strategy tends to be largely tactical in nature, with ideas generated on an ad-hoc basis and experiment prioritisation based on most urgent priority or a simple impact/ease model. Each experiment is treated as an individual exercise.

Advanced optimisation teams plan their strategy for achieving their optimisation goals across both the short-term and long-term. Long-term strategic planning should focus on prioritisation at the high level of goals and priorities. Conversion optimisation is an ongoing process. It’s not possible to do everything at once, and mature teams plan and prioritise the areas that they will focus on right now and those that they’ll focus on later in the year. In this way they can keep their focus narrow and ensure there is a clear plan for achieving their goals.

Advanced optimisation teams view testing not as a tool for increasing conversion rates but as a tool for answering questions. Starting with the big picture, they identify the business questions that need to be answered. They then break these problems down to define the tests and research that they need to complete to validate their hypotheses and answer that question.

As we move up the maturity stages, optimisation strategy becomes more thematic. Experiments are considered now as one tool for exploring a specific theme or conversion lever. At this level, experimentation is organised as a series of projects, each made up of a combination of targeted user-research pieces and experiments. These projects align to business strategy, and experimentation starts to play a leading role in overall business strategy.  

Data & technology strategy

How do you detect and measure the things that matter?
The quality of insight gained from experimentation is directly correlated to the quality of data that you collect about what happened. If your goal is just to get some experiments live there is probably less emphasis on ensuring those experiments have a solid grounding in data. Ensuring the data the experiments produce when they do run is reliable and actionable can often be more of an afterthought. Advanced optimisation teams will be a lot more deliberate, with data and insight playing leading roles in generating test hypotheses, and experiment data being a valuable source of insight for the business and the people in it. Maturity here is being confident in your data so that you can challenge it, ask probing questions of experiment impact, and be able to confidently produce the answers.

Technology plays a key role in this, but is only as good as the strategy for using it. The specific tools you use aren’t as important, for example, as your ability to connect your tools and data sets together. A set of simple but connected tools can deliver greater quality of insight that one advanced but isolated tool. Start with your experimentation tool, and connect it to any other tools you have such as surveys, session recording and heatmaps. In particular, connect it to your back-end reporting systems so that the impact of experiments can be measured against the KPIs that really matter, and that people look at on a daily basis.

Maturity levels and where you place

Now that we’ve explored the 3 scales that we use to measure maturity we can define approximate levels of maturity to give us an overall scale and tool for evaluating our own place. Really though, maturity is a continuous scale rather than something discreetly split into levels. When reviewing the levels below you may place yourself at different levels for each of the 3 scales. This is very common. There is often one part of our approach that we know is probably holding us back – a weak link in the chain. This model should help formalise and pinpoint that weakness and start the conversations for how to overcome it.

If you’re looking to develop the maturity of your experimentation and conversion optimisation strategy then we’d be happy to help. Just drop an email to hello@conversion.com and we’ll organise a free maturity consultation with one of our team.