The Optimizely Customer Workshop: How do the UK’s biggest brands approach experimentation?

Go back to blog

Pin It

The Optimizely Customer Workshop, hosted by Phil Nayna (Enterprise Account Executive at Optimizely) and Stephen Pavlovich (Founder/CEO of Conversion.com), brought together representatives from some of the UK’s biggest brands to share their thoughts and insights on Conversion Rate Optimisation (CRO). The workshop took shape in the form of a roundtable where talk topics included: “Building a lean testing programme”, “Applying testing to business challenges” and, the buzzword of the moment, “Personalisation”.

 

Building a lean testing programme

Experience in testing and experimentation amongst attendees in the room ranged from businesses who were just starting out, to those that had already produced mature testing programmes. This range of experience provided the basis for a profound discussion. For UK brands just starting their testing, it was emphasised that obtaining buy-in from stakeholders was key to building a testing programme within their companies. For brands with more testing experience, the biggest challenge in building this lean programme came with shifting their culture. Key to adopting a testing culture is acknowledging – and leveraging – the focus on short-term testing and validation over long-term planning. That’s why the attendees all agreed that a short-term iterative roadmap is far better than a long-term rigid roadmap.

Here at Conversion.com, experimentation is at the heart of everything we do and who we are. We believe that building a lean testing programme and cultivating a testing culture relies on two key factors: education and sharing. Educating your employees to understand your philosophy on experimentation and its benefits is key. This allows your employees to view experimentation as far more than just the potential value it yields with winning tests. At Conversion.com we value education highly. We run our own CRO training programme for new associate consultants that educates them and allows them to think creatively and with ambition when it comes to experimentation and CRO. When sharing experiment results with clients, it’s crucial not just to share what was tested and what the results were, but more importantly why we tested it and what it can teach us about their users. This means that with every experiment, we learn more about their users, allowing us to refine and improve our testing strategy – while delivering measurable uplift.

 

Applying testing to business challenges (prioritisation of your testing roadmap)

Strategies for prioritising testing roadmaps varied extensively within the workshop, with all brands favouring a different approach or primary metric. One major UK supermarket brand stated that their approach was very data-driven, something we value highly at Conversion.com. They prioritised ease of implementation, lack of organisational friction in getting the test launched, potential impact the test has and the data or evidence supporting the hypothesis. Other primary metrics included cost impacts, due to one UK brand having a lack of development resource. This meant they favoured the ease of the test as a priority, as it allowed them to test despite this barrier.  

At Conversion.com we believe that the data driving a test is most important when prioritising our tests. This data informs us of the impact that the test is likely to have. Secondary metrics, such as the ease of building the test and getting sign-off  – as well as the other tests and hypotheses we have running in parallel – allow us to see how and when this test fits into our roadmap. However, it is important to note that prioritisation can be limited. There are finite swimlanes to test and finite resources, meaning prioritisation and planning have to be coherent. Understanding that testing roadmaps have to be flexible and adaptive is key. This allows us to easily change our roadmap according to the performance of previous tests and as our understanding of users improves.

 

Personalisation

Personalisation is the buzzword of the moment in CRO and this topic divided our workshop audience. Some UK brands stated that they had banned the word completely. Instead they refer to this as creating more relevant customer experiences and concentrating on more targeted journeys. All representatives agreed that their personalisation journey was at its early stages, believing it was important to keep personalisation simple and start getting tests live in order to gain momentum. However, we believe that this could increase the risk of companies starting personalisation too early and as a result, missing valuable opportunities for increasing their conversion rate with all-audience A/B testing. With personalisation being such a hot topic, it is critical that companies take the time to integrate this into their wider digital strategies as opposed to implementing it without consideration for other key areas of CRO.

At Conversion.com, we view personalisation as optimising conversion by increasing the relevance of experiences for specific audiences. Although we see personalisation as a great and exciting new opportunity to test, we believe it is important to successfully assess when to start personalisation. By its nature, it forces you to focus on a subset of users, potentially diminishing the impact of experiments as well as complicating future all-audience experiments.

 

The Optimizely Customer Workshop was the perfect setting for valuable discussions and an insight into how the UK’s biggest brands approach experimentation. From the workshop the key takeaways were:

  • Education of CRO needs to be more highly regarded within businesses in order to promote a shift in testing culture.
  • Visibility of testing programmes via sharing of content allows employees to understand the value of testing past just the potential value of winning tests.
  • Roadmaps should be as flexible and adaptive as possible to allow for test and learn iterations to occur.
  • Personalisation should be undertaken when its potential overtakes all-audience testing and should integrate with – rather than replace – typical A/B testing for CRO.

July 18th, 2017

About the Author

 

Sadie Neve

Sadie is an Associate Consultant at Conversion.com. She has a background in Psychology and a keen interest for both qualitative and quantitative insights.

Comments (0)

There are no comments.

Post your comment here

Get involved in the conversation! Leave a comment and we or someone else in the community will be sure to reply.

 
 

Get in touch

If you're looking for a personal, results-focused approach to conversion optimization, we'd love to hear from you.

Get In Touch

For the best experience, please return your device to portrait mode.