Conversion.com hosts... 'Experimentation Maturity: What advanced testing teams do differently'

We are very excited about the release of our Ecommerce Performance Report for 2018 in partnership with Econsultancy. The report covers various concepts in-depth ranging from the growth of the ecommerce market to the future of experimentation however, one thing got us talking at HQ – experimentation maturity.

Out of the 400 ecommerce professionals surveyed, 50% stated that they perceived the value of experimentation to be high/very high however, only 14% stated that their business recognised it as a strategic priority. The disparity between value and strategic prioritization got us thinking about the roadblocks to experimentation maturity and how we can overcome them.

Kyle Hearnshaw, Head of Conversion Strategy, Stephen Pavlovich, our CEO and James Gray, Senior Optimisation Manager at Just Eat, led our second independent event where we brought practitioners from the industry together to discuss five key themes around experimentation maturity:

1. What defines a mature approach to experimentation and conversion optimization?
2. How can you measure your growth in maturity over time?
3. How does your organization stack up compared to 450+ respondents in our report?
4. What challenges do we face in developing maturity?
5. How can you overcome these challenges?

Measuring experimentation

In the past we have seen organizations use the size of the experimentation team, the number of experiments launched or the complexity of experimentation as metrics to measure their experimentation maturity.

At Conversion.com, we believe such metrics should be avoided and that maturity should be measured against quality. This means moving the goalposts so that we are benchmarking against experimentation goals, experimentation strategy and data and technology strategy.

What are you trying to achieve via experimentation? 

Setting goals in which to measure the success of experimentation is pivotal for any organization. At Conversion.com, we recommend following three steps to develop robust experimentation goals:

1. List your key business challenges – the most mature experimentation programs drive the strategic direction of businesses. If you feel a long way off this, don’t panic. Listing business challenges and making sure experiments have a measurable impact against these is a huge step on the journey to maturity.
2. Set a specific goal for experimentation – individual experiments need specific goals and KPIs. In order to translate business challenges into specific experimentation goals we recommend using a goal tree map.
3. Plan a roadmap to develop maturity – recognising your business’ position on the maturity scale is important in identifying necessary steps to reach maturity. At Conversion.com, we have created an experimentation maturity assessment so you can easily plot where you sit on the scale; be honest, a realistic benchmark allows you to identify the key steps to take in order to progress.

How do you organise and deliver experimentation? 

Experimentation strategy often gets forgotten about, as a consequence holistic experimentation can be derailed and the link between business strategy and goals can break.

In order to prevent this disconnection we recommend that you:

1. Set regular points to review strategy – stand back at regular intervals to look at the big picture. Involve stakeholders from across the business and brainstorm strategic priorities.
2. Organise experiments into projects/themes – Ad hoc tactical experiments can be chaotic therefore, we recommend defining projects that group related research, experiments and iterations. This allows goals and outcomes to be measured at a project level.
3. 10x your communication – The goal here is to get more people involved in experimentation. Some great examples were shared at our event – Just Eat talked us through their approach to ensuring experimentation strategy is shared and reviewed across areas of the business with their experimentation forum. Within this forum every product manager pitches their experiments, detailing what they’d like to test and how they plan to measure the results. We think this is an excellent way to create open communication across a company.

How do you measure the impact of experimentation?

Evaluating the impact of experimentation is pivotal in order to determine the success of testing. But what measurements are important?

1. Define standards on experiment data – at Conversion.com, we talk about ‘North Star’ metrics. Experiments that have an impact on these metrics should get noticed by senior stakeholders. This tactic resonated with those at our maturity event – organizations voiced that they struggled to get executive buy-in without proving value through results.
2. Strive for a single customer view – a single customer view is an aggregated, uniform and comprehensive representation of customer data and behavior. Going beyond single conversion metrics is a huge leap in the path to maturity however, it is not easily achieved. We recommend integrating testing tools with business intelligence tools in order to gather data such as lifetime value.
3. Build in segmentation as early as possible – no matter where you are on the maturity scale we highly recommend identifying key audiences in order to lay the foundations for personalization. With this in place, you can report on audience behaviors within experiments and uncover greater insights from your experiments.

So, what next? 

Identify your organization’s maturity level using our maturity model. Whether you are just getting started or are an advanced team, there are actions you can take to reach the highest levels of maturity.

We thoroughly enjoyed hosting our second independent event. Insightful discussions were had across a variety of organizations and we are proud to have been able to offer advice to assist organizations on progressing towards their own experimentation maturity.

If you’d like to attend future events, keep an eye on our events page.