Brands increasingly understand the importance of experimentation to constantly improve and optimise their digital performance.
However, creating this culture of experimentation, where failure is seen as something to learn from can be difficult. How brands can build this culture and drive improved performance, was the focus of our recent joint event with Kameleoon at Sea Containers House.
Exploring the topic with a packed room were James Gray of Facebook, Marianne Stjernvall of TUI and Conversion’s Stephen Pavlovich. In a wide ranging discussion that moved from the 18th to the 21st centuries, and from finding the cure to scurvy to the Amazon Echo, the speakers explained hints, tips and case studies for building a strong experimentation culture.
Experiments that lose, build a better business case than the ones that win!
Starting the evening, Stephen Pavlovich explained his best practices for experimentation, starting with the importance of recognising and celebrating failure, rather than simply sweeping it under the carpet.
Too many organisations only talk about winning experiments, rather than learning from the losers. This leads to poor practice as losing experiments can often outweigh the value of the winning ones. Take Amazon’s disastrous Fire smartphone, which failed to sell and cost the company $170 million. Much of the technology was repackaged into the Amazon Echo smart speaker, creating an incredibly successful product and channel for the company.
Next Stephen addressed the issue of getting buy-in for experimentation, either by scaling experiments up to address bigger issues (push), or scaling something big down to make it more manageable (pull). This helps spread experimentation across the business and directly supports a culture where more radical ideas are tested. As he concluded, “We shouldn’t just test what we were going to do anyway. We should experiment on our boldest ideas – with a safety net.”
Getting it wrong…
Facebook is often highlighted as a leader in experimentation. However, as James Gray, Growth Marketing Manager in Facebook’s Community Integrity division explained, you still need to follow a structured and sensible process in order to drive optimal results. He highlighted three areas to focus on:
- Focus on the basics: Make sure that experiments are run for long enough to deliver realistic results (two business cycles is a logical timeframe), that you are choosing (and sharing) the right metrics to measure the experiment BEFORE it starts and that you have a process to widely share the results.
- Adapt to your terrain: Every business is different, so make sure your experiments reflect the idiosyncrasies of your organisation. Cater for factors that might either impact your results or mean that results don’t always translate into the promised benefits down the line.
- Navigate your organisation: To get your results accepted and acted upon you need to bring the rest of the organisation with you. That means winning people over by taking the time to explain what you are doing and why your results are important. James shared the case of 18th century doctor John Lind who tested different potential treatments for scurvy amongst sailors. While his experiment successfully identified citrus fruit as the cure, the naval authorities took 42 years to put this finding into practice – primarily because it was not clearly communicated in ways that these non-specialists could understand.
Creating an experimentation culture at TUI
Large, traditional businesses often find it hard to change to a culture of experimentation, held back by organisational structures and a more conservative way of operating. Demonstrating that this doesn’t need to be the case, Marianne Stjernvall, CRO Lead at TUI, outlined six ways to get your culture of experimentation off the ground, based on her experience at the international travel group.
- Be the hero: Have a clear plan on what you want to do, introduce the idea of testing, and listen to stakeholders to get them onside. Remember that humans are hard-wired to be suspicious of change, so not everyone will be positive from the start.
- Run masterclasses: Show people what you are passionate about with regular high-level, fun sessions that are open to all. Keep running them and you’ll draw in more people from across the business that you otherwise would never get to speak to.
- Share the results: Hold meetings with key stakeholders to drill down into your results. Be sure they understand the data and what it means before increasing the level of your detail and transparency through tools such as dashboards.
- What I Learnt This Week: Weekly team meetings where everyone has the chance to share what they have done, and what has resulted from it. Make it user and people focused.
- CRO Showcase: Highlight every learning through regular calls with your team, particularly if you are spread geographically in different countries.
- Monthly dashboard: Introduce an element of (friendly) competition by not only reporting on all tests and their impact on business metrics, but highlighting the test that performance best of all. What can be learned and applied elsewhere?
The evening finished with an interactive panel Q&A, covering topics as diverse as “most embarrassing test”, to what to do if your test proves inconclusive.
If you’d like to learn more about how Conversion can help you use experimentation to drive growth in your business, get in touch today. If you’d like to find out more about Kameleoon’s A/B testing and experimentation platform, then visit their website here.
