Author: Josh D'arcy

Brands increasingly understand the importance of experimentation to constantly improve and optimise their digital performance.

However, creating this culture of experimentation, where failure is seen as something to learn from can be difficult. How brands can build this culture and drive improved performance, was the focus of our recent joint event with Kameleoon at Sea Containers House.

Exploring the topic with a packed room were James Gray of Facebook, Marianne Stjernvall of TUI and Conversion’s Stephen Pavlovich. In a wide ranging discussion that moved from the 18th to the 21st centuries, and from finding the cure to scurvy to the Amazon Echo, the speakers explained hints, tips and case studies for building a strong experimentation culture.

Experiments that lose, build a better business case than the ones that win!

Starting the evening, Stephen Pavlovich explained his best practices for experimentation, starting with the importance of recognising and celebrating failure, rather than simply sweeping it under the carpet.

Too many organisations only talk about winning experiments, rather than learning from the losers. This leads to poor practice as losing experiments can often outweigh the value of the winning ones. Take Amazon’s disastrous Fire smartphone, which failed to sell and cost the company $170 million. Much of the technology was repackaged into the Amazon Echo smart speaker, creating an incredibly successful product and channel for the company.

Next Stephen addressed the issue of getting buy-in for experimentation, either by scaling experiments up to address bigger issues (push), or scaling something big down to make it more manageable (pull). This helps spread experimentation across the business and directly supports a culture where more radical ideas are tested. As he concluded, “We shouldn’t just test what we were going to do anyway. We should experiment on our boldest ideas – with a safety net.”

Getting it wrong…

Facebook is often highlighted as a leader in experimentation. However, as James Gray, Growth Marketing Manager in Facebook’s Community Integrity division explained, you still need to follow a structured and sensible process in order to drive optimal results. He highlighted three areas to focus on:

1. Focus on the basics: Make sure that experiments are run for long enough to deliver realistic results (two business cycles is a logical timeframe), that you are choosing (and sharing) the right metrics to measure the experiment BEFORE it starts and that you have a process to widely share the results.
2. Adapt to your terrain: Every business is different, so make sure your experiments reflect the idiosyncrasies of your organisation. Cater for factors that might either impact your results or mean that results don’t always translate into the promised benefits down the line.
3. Navigate your organisation: To get your results accepted and acted upon you need to bring the rest of the organisation with you. That means winning people over by taking the time to explain what you are doing and why your results are important. James shared the case of 18th century doctor John Lind who tested different potential treatments for scurvy amongst sailors. While his experiment successfully identified citrus fruit as the cure, the naval authorities took 42 years to put this finding into practice – primarily because it was not clearly communicated in ways that these non-specialists could understand.

Creating an experimentation culture at TUI

Large, traditional businesses often find it hard to change to a culture of experimentation, held back by organisational structures and a more conservative way of operating. Demonstrating that this doesn’t need to be the case, Marianne Stjernvall, CRO Lead at TUI, outlined six ways to get your culture of experimentation off the ground, based on her experience at the international travel group.

1. Be the hero: Have a clear plan on what you want to do, introduce the idea of testing, and listen to stakeholders to get them onside. Remember that humans are hard-wired to be suspicious of change, so not everyone will be positive from the start.
2. Run masterclasses: Show people what you are passionate about with regular high-level, fun sessions that are open to all. Keep running them and you’ll draw in more people from across the business that you otherwise would never get to speak to.
3. Share the results: Hold meetings with key stakeholders to drill down into your results. Be sure they understand the data and what it means before increasing the level of your detail and transparency through tools such as dashboards.
4. What I Learnt This Week: Weekly team meetings where everyone has the chance to share what they have done, and what has resulted from it. Make it user and people focused.
5. CRO Showcase: Highlight every learning through regular calls with your team, particularly if you are spread geographically in different countries.
6. Monthly dashboard: Introduce an element of (friendly) competition by not only reporting on all tests and their impact on business metrics, but highlighting the test that performance best of all. What can be learned and applied elsewhere?

The evening finished with an interactive panel Q&A, covering topics as diverse as “most embarrassing test”, to what to do if your test proves inconclusive.

If you’d like to learn more about how Conversion can help you use experimentation to drive growth in your business, get in touch today. If you’d like to find out more about Kameleoon’s A/B testing and experimentation platform, then visit their website here.

Recently we held, what we certainly thought, was our best event yet. An evening of experimentation co-hosted by our friends at Facebook.

We were lucky enough to be joined by Vince Darley (Head of Growth at Deliveroo), Brian Hale (Vice President of Growth Marketing at Facebook) and Denise Moreno (Director of Growth Marketing at Facebook), who shared some incredible insights.

Held at the Facebook offices in Rathbone Square, London, we welcomed a variety of guests from top brands across an array of industries, to hear all about the key principles of experimentation.

Proceedings began with our very own Stephen Pavlovich, CEO and Founder of Conversion.com. If you’re working in the marketing or ecommerce space, it’s likely that you’ve come across the basic principles of experimentation. However, Stephen wanted to explain what experimentation REALLY is.

It’s not as simple as just running a few A/B tests on your site to find out which button colour converts the best, but experimentation, once taken to place where your business has reached an advanced level of maturity, can really begin to answer the wider, more important questions. For example, which product should you launch next? Or how should you structure your commercial model? We believe that experimentation really should be at the heart of every single business.

Next up, we had Vince Darley, Head of Growth at Deliveroo. Vince has had years of experience leading experimentation teams at huge businesses like King and Ocado. He wanted to share some of the knowledge and experience he’s gained over many years at the top with our guests.

Vince really showed the breadth of his knowledge by making sure there was something there for everybody through his ‘Experimentation Three-Course Meal’. He began by sharing some of the basic rules for anybody getting started with experimentation, his experience at King working on applications like the formidable Candy Crush, sharing some of the most important lessons he has learnt including the best way to conduct high impact experimentation. Finally, he drew on his expert knowledge to share some of his most advanced secrets for experimentation, allowing the audience to leave with some indispensable tips around using and interpreting data.

With a tough act to follow, next our audience was treated to a double act by Brian Hale, Vice President of Growth Marketing, and Denise Moreno, Director of Growth Marketing at Facebook.

They discussed five key issues around creating a growth team and how experimentation should inform the process and ways of working. It was incredible for our audience to hear about how one of the most iconic growth teams in the world was formed. They were provided with actionable insight that will help them when they try to drive the expansion of their own internal teams. Finally, they shared some of their own experimentation principles from years of experience, and some real life examples from Facebook around testing on messenger and ads from some of the earliest stages of the platform.

Alongside some fantastic content from our speakers, the evening also marked the launch of our experimentation principles project. Inspired by the simple elegance of the UK government design principles, we have decided to collate our 11 years experience at Conversion.com, and define a set of core experimentation principles. They tackle the simple mistakes, misconceptions and misinterpretations that organisations make, that limit the impact, effectiveness and adoption of experimentation.

Every member of the audience received a copy of these to help elevate their experimentation programme. Luckily, this wasn’t limited to our guests, and you too can elevate your programme. If you’d like to download a copy of the key principles you should be considering, with input from our friends at Facebook, Just Eat, Booking.com and Microsoft, then download your copy here today.

If you’d like to hear more about how you can use experimentation to drive growth in your business, then get in touch.

Keep an eye on our events page to make sure you don’t miss out. 

We recently kicked off a workshop series with one of our solution partners, Optimizely, starting proceedings off (quite fittingly) with a session on the topic of ideation.

The workshop session was led by Kyle Hearnshaw, Head of Conversion Strategy at Conversion.com, with support from Stephen Pavlovich, CEO and Nils Van Kleef, Solutions Engineer at Optimizely.

We run ideation sessions all the time, whether between ourselves or in collaboration with our clients. But this had a slightly different twist. We gathered a group of representatives, from multiple different businesses, across multiple different verticals, in a room to experiment with how well different approaches to ideation perform. Every attendee left with an understanding of the different approaches, and insight into which approach they felt they could apply most effectively with their own teams.

For those of you that couldn’t make the session, we want to share some of those ideas so that you too can run ideation sessions that generate impactful ideas.

But first, why does the ideation process matter?

Ideation injects creativity into a data-driven process

Data is incredibly important in driving all of the decisions we make in experimentation. Data is great at telling us where the problems are, but it isn’t good at telling us how to solve these problems. This is where creativity plays a key role in experimentation. Ideation is our opportunity to inject creativity into what we do, to explore new concepts and experiment with potential solutions as we hone in on the optimal solution for each problem.

Our time is limited

For many of us, there’s barely enough hours in the day to get through all our emails, let alone spend hours in unproductive brainstorming. We need to optimise how we spend our time in ideation to get a good return on our time investment. The ideation process needs to reliably generate high quality ideas that can immediately be tested through experimentation.

More people doesn’t always mean better ideas

We’ve been taught that collaboration should generate more and better ideas, but this is only true when people can contribute effectively. When more people are in the room, there is often a tendency for people to become protective of their own ideas, as opposed to sharing, discussing and letting their ideas evolve through the input of the rest of the group. Everyone present in an ideation session should be able to, and also expected to contribute effectively.

Subtle differences in execution can have a big impact on results and build time

The difference between an execution of an experiment that wins versus an execution that loses can be extremely small. Two ideas that seem similar can often perform very differently when exposed to real users. One way of testing an idea might require a lot more effort to create than another. Ideation plays a key role in both defining and refining ideas.

Our top 4 tips for effective ideation, no matter your approach

1. Separate the hypothesis and the execution 

Ideas that come up in an ideation session come in all shapes and sizes. Some will come to you fully formed with a well-defined hypothesis and a sketch of the execution of the experiment. Others will be less-formed, and need to be refined into a clear hypothesis first before ideating around the best execution of that idea. When an idea is raised, identify whether it’s a hypothesis or a specific execution. For hypotheses, you can then ideate on solutions. For executions you can step back and ideate around the underlying hypothesis of the idea.

2. Break ideas up into sequences of experiments – starting with the minimum viable experiment

One way of doing this is to think about how the idea might fit into a sequence of experiments. Think about the idea you want to explore. Is it the ‘minimum viable experiment’? Or is it a more formed exploration of a specific solution in this area? Think about how you might take your idea and iterate on it across a range of experiments, continuing to improve, and reap results of the idea you have produced to finally reach that ‘perfect’ version.

3. Is this idea iterative, innovative or disruptive?

When you reach that stage of an ideation session where the ideas start to dry up, a useful exercise is to group your ideas into 3 types. Which are iterative ideas – tweaks, optimisations or designs? Which are innovative ideas – new experiences, journeys or usability? And which are groundbreaking, disruptive ideas that will affect product, pricing, or even the company proposition? Try and make sure you have a good number of each category.

4. Have a plan to weed out your bad ideas

Remember back at school, when you were told to put your hand up and express your ideas, whatever they were? There’s no such thing as a bad idea, right? Wrong.

Contrary to common belief, there is such thing as a bad idea. We would encourage you to have a completely open system for idea creation. Allow people to come up with whatever ideas they like. But, also have a system for critique and review. Some ideas will simply not be good, others will be good, but won’t be possible.

There has to be a good mix of realistic, and ambitious ideas. The last thing anybody wants to do is waste their precious time talking about ideas that are simply ‘dead in the water’. One approach we like for this is using The Disney Method.

The 3 approaches to ideation we tested

We’re going to share with you the three approaches that we explored in our ideation workshop.

Unstructured Ideation

Quite often when we talk to people and ask how they ideate, the answer is “Well, we get everyone in a room and talk about some ideas”. This is the more common type of ideation session, and by no means the approach that we would recommend. But, it does provide a useful baseline, and if you apply all of our tips above it can, in the right circumstances, still be effective.

That said, it’s pretty much a free-for-all. Everyone shouts their ideas, there is very little focus, it’s unstructured. For this session, we gave attendees a vague goal of increasing conversion rate and a specific page of a website to improve – to mimic the setup of one of these sessions in a conversion optimisation context.

Pros: 

+ Anywhere, anytime

+ Anyone can do it

Cons:

– Anyone can do it (the input expected from each attendee isn’t clear)

– The ideas generated tend to be unrelated and broad

– Small number of high-quality ideas

Structured Ideation

In most cases, when we talk about ideation we’re talking about the creation of ideas for AB tests and experiments. What many people fail to remember, is that experiments are just the end-product of a larger strategic process. At Conversion, we build our strategy around our experimentation framework.

Most unstructured ideation sessions tend to be around a loosely defined goal and perhaps a KPI to be improved. However, in order to conduct an effective ideation session we need more structure and focus.

Only after you have defined your Goal and KPIs, then used data to understand and define your Audiences, Areas and Levers, should you start to ideate for experiment concepts. Agree the specific audience, area and lever that you’re going to ideate on and make sure everyone knows this, and has seen any relevant data and research before they attend. The session will then be more focused around solving a specific problem, and structured.

For this session we gave attendees a completed experimentation framework and defined the Goal, KPIs, Audience, Area and Lever we wanted them to ideate on.

Pros:

+ Customer-focused

+ Impactful concepts

Cons:

– Needs upfront research

– Takes longer to get started

Crazy 8s

This is one of our favourite and most enjoyable methods of ideation. It forces every member of the group to produce 8 ideas, in 8 minutes. We’ve adapted the concept from one originally adopted by Google.

Our adaptation of the original Crazy 8s system is to apply it to a structured ideation setup. So again we have defined our Goal, KPIs, Audience, Area and Lever. Then we use the Crazy 8s ideation process on that specific lever to generate a large number of ambitious ideas in a short space of time. Rather than one person generating 8 ideas on one lever, we often rotate the paper so that you have 1 minute to add a new idea on a lever that nobody else has come up with yet. In this way you can cover multiple levers in one session if you’re ambitious.

The purpose of using Crazy 8s is to force everyone in the ideation session to contribute, but also to stretch all of the attendees to contribute more ambitious, creative ideas then they might generate without the added time pressure. It also encourages people to draw ideas to save time, which can bring out new ideas as people get more visual.

Pros:

+ Large number of ideas

+ Includes all the structured ideation benefits

Cons:

– Needs an introduction

– Less collaborative

Our conclusions

Overall, we thought the workshop was a great success. We had some great feedback from those involved and some brilliant ideas for our attendees to take away.

The key takeaway was that structure is crucial for effective ideation. And by that we don’t mean the minute-by-minute structure of the session itself, we mean the structure, focus and setup of what you’re ideating about. A structured approach not only generates more ideas, but generates crucially more impactful, creative and ambitious ideas. As the host of the session you will walk away with both the quantity and quality of ideas you need to design your experiments.

Ideation is critical to experimentation. In order to create an effective experimentation roadmap, you must engage in effective ideation. Following just a few of these techniques, will have you well on your way. But of course, if you’d like to know more, do get in touch.

If you’d like to attend future events, keep an eye on our events page.