Events Archives | Conversion.com

Talking PIE over breakfast – our prioritisation workshop

Recently, we continued our workshop series with one of our solutions partners, Optimizely, discussing the prioritisation of experiments.

The workshop session was led by Kyle Hearnshaw, Head of Conversion Strategy at Conversion.com, with support from Stephen Pavlovich, CEO and Nils Van Kleef, Solutions Engineer at Optimizely.

Our most popular workshop to date, we gathered over 40 ecommerce professionals including representatives from brands such as EE, John Lewis and Just Eat together, all keen to talk about one of their biggest challenges – prioritisation. Throughout the morning we discussed why we prioritise, popular prioritisation methods and finally how we at Conversion.com prioritise experiments.

For those of you that couldn’t make the session, we want to share some insights into prioritisation so you too can apply learnings next time you are challenged with prioritising experiments. Keep an eye out on our blog too, as later in the year we’ll be posting a longer step-by-step explanation of our approach to personalisation.

Why Prioritise?

There is never usually a shortage of ideas to test however we are often faced with a shortage of resources to build, run and analyse experiments as well as traffic in which to run experiments on. We need to make sure we prioritise the experiments that are going to do the most to help us achieve our goal in the shortest of time.

So, what is the solution?

Popular Prioritisation Methods

In order to identify those tests that have the maximum impact with the efficient use of resources we need to find the most effective prioritisation method. So, let’s take a look at what is out there:

1. PIE model

Potential: How much improvement can be made on the pages?

Importance: How valuable is the traffic to the pages?

Ease: How complicated will the test be to implement on page / template?

We think PIE is simple and easy to use, analysing only 3 factors. However, some concerns with this model are  that the scoring can be very subjective and there can be an overlap between Potential and Importance.

2. Idea Scores from Optimizely 

 Optimizely’s method is an extended version of the PIE model adding the factor of ‘love’ into the equation. Again, we commend this model for its simplicity however, subjectivity from scoring can still mean the overall score is subjective.

3. PXL model from ConversionXL

 The PXL is a lot more complex than the previous two, giving eight to data and insight which we think is very important. In addition, the PXL model goes a way towards eliminating subjectivity by limiting scoring to either 1 or 0 in most columns. One limitation of this model is that it doesn’t account for differences in page value rather than just page traffic, nor does it give you a way to factor in learnings from past experiments. It also has the potential to be very time consuming and you may not easily be able to complete all columns for every experiment.

Prioritisation at Conversion

When deciding on our prioritisation model we wanted to ensure that we were prioritising the right experiments, making sure the model accounted for insights and results, removing any possibility for subjectivity and allowing for the practicalities of running an experimentation programme. So, we came up with the SCORE model:

The biggest difference with our approach is that prioritisation happens at two separate stages. We want to avoid a situation where we are trying to prioritise a large number of experiments with different hypotheses, KPIs, target audiences and target pages against each other. In our approach individual experiments are prioritised at the ‘Order’ stage however, we minimise the need for directly prioritising experiments against each other by first prioritising at the strategy stage.

We use our experimentation framework to build our strategy by defining a goal, agreeing our KPIs and then by prioritising relevant audiences, areas and levers. Potential audiences we can experiment on are prioritised on volume, value and influence. Potential areas are prioritised on volume, value and potential. Levers – what user-research has show could influence user behaviour – are prioritised on win rate (if we’ve run experiments on this lever before), confidence (how well supported the lever is in our data) or both.

Next we ensure we cultivate the right ideas for our concepts. We believe structured ideation around a single hypothesis generates better ideas. Again, utilising the experimentation framework we define our hypotheses: “We believe lever for audience, on area will impact KPI.” Once the hypothesis has been defined we then brainstorm the execution.

The order of our experiments come from prioritising the concepts that come out of our ideation sessions. Concepts can be validated quickly by running minimum viable experiments. MVEs allow us to test concepts without over-investing and also allows us to test more hypotheses in a shorter timeframe.

Next, we create an effective roadmap. We start by identifying the main swimlanes (pairs of audiences and areas that can support experiments) and then we estimate experiment duration based on a minimum detectable effect. A roadmap should include tests across multiple levers, this allows you to gather more insight and spreads the risk of over-emphasising in one area.

Finally, it’s time to run and analyse the experiments (execution).

We believe our SCORE model is effective for prioritising experimentation projects because it puts more emphasis on prioritising and getting the right strategy first before ever trying to prioritise experiments against each other. It is structured, rewards data and insight and allows for the practicalities of experimentation – we can review and update our strategy as new data comes in. The only limitation is that it does take time in order to prioritise the strategy effectively. But, if we’re going to invest time anywhere we believe it should be on getting the strategy right.

Our conclusions

The workshop was a great success. We had some great feedback from those involved and some actionable ideas for our attendees to take away.

We recommend having a go at using the SCORE prioritisation model. In the next few weeks we’ll be sharing a detailed post on our experimentation framework but you can apply SCORE within your own approach by reviewing how you define and prioritise your experimentation strategy. See whether this helps you to produce a roadmap which is informed by data and insight, absent of subjectivity and effective in helping your business test the most valuable ideas first.

If you have any questions or would like to know more, please get in touch.

To attend our future events, keep an eye out here.

An evening of experimentation…with Facebook and Conversion.com

Recently we held, what we certainly thought, was our best event yet. An evening of experimentation co-hosted by our friends at Facebook.

We were lucky enough to be joined by Vince Darley (Head of Growth at Deliveroo), Brian Hale (Vice President of Growth Marketing at Facebook) and Denise Moreno (Director of Growth Marketing at Facebook), who shared some incredible insights.

Held at the Facebook offices in Rathbone Square, London, we welcomed a variety of guests from top brands across an array of industries, to hear all about the key principles of experimentation.

Proceedings began with our very own Stephen Pavlovich, CEO and Founder of Conversion.com. If you’re working in the marketing or ecommerce space, it’s likely that you’ve come across the basic principles of experimentation. However, Stephen wanted to explain what experimentation REALLY is.

It’s not as simple as just running a few A/B tests on your site to find out which button colour converts the best, but experimentation, once taken to place where your business has reached an advanced level of maturity, can really begin to answer the wider, more important questions. For example, which product should you launch next? Or how should you structure your commercial model? We believe that experimentation really should be at the heart of every single business.

Next up, we had Vince Darley, Head of Growth at Deliveroo. Vince has had years of experience leading experimentation teams at huge businesses like King and Ocado. He wanted to share some of the knowledge and experience he’s gained over many years at the top with our guests.

Vince Darley, Head of Growth at Deliveroo
Vince Darley, Head of Growth at Deliveroo

Vince really showed the breadth of his knowledge by making sure there was something there for everybody through his ‘Experimentation Three-Course Meal’. He began by sharing some of the basic rules for anybody getting started with experimentation, his experience at King working on applications like the formidable Candy Crush, sharing some of the most important lessons he has learnt including the best way to conduct high impact experimentation. Finally, he drew on his expert knowledge to share some of his most advanced secrets for experimentation, allowing the audience to leave with some indispensable tips around using and interpreting data.

With a tough act to follow, next our audience was treated to a double act by Brian Hale, Vice President of Growth Marketing, and Denise Moreno, Director of Growth Marketing at Facebook.

Brian Hale, Vice President of Growth Marketing at Facebook
Brian Hale, Vice President of Growth Marketing at Facebook

They discussed five key issues around creating a growth team and how experimentation should inform the process and ways of working. It was incredible for our audience to hear about how one of the most iconic growth teams in the world was formed. They were provided with actionable insight that will help them when they try to drive the expansion of their own internal teams. Finally, they shared some of their own experimentation principles from years of experience, and some real life examples from Facebook around testing on messenger and ads from some of the earliest stages of the platform.

Alongside some fantastic content from our speakers, the evening also marked the launch of our experimentation principles project. Inspired by the simple elegance of the UK government design principles, we have decided to collate our 11 years experience at Conversion.com, and define a set of core experimentation principles. They tackle the simple mistakes, misconceptions and misinterpretations that organisations make, that limit the impact, effectiveness and adoption of experimentation.

Our 9 key principles of experimentation
Our 9 key principles of experimentation

 

Every member of the audience received a copy of these to help elevate their experimentation programme. Luckily, this wasn’t limited to our guests, and you too can elevate your programme. If you’d like to download a copy of the key principles you should be considering, with input from our friends at Facebook, Just Eat, Booking.com and Microsoft, then download your copy here today.

If you’d like to hear more about how you can use experimentation to drive growth in your business, then get in touch.

Keep an eye on our events page to make sure you don’t miss out. 

How to make your ideation sessions go down a (brain)storm

We recently kicked off a workshop series with one of our solution partners, Optimizely, starting proceedings off (quite fittingly) with a session on the topic of ideation.

The workshop session was led by Kyle Hearnshaw, Head of Conversion Strategy at Conversion.com, with support from Stephen Pavlovich, CEO and Nils Van Kleef, Solutions Engineer at Optimizely.

We run ideation sessions all the time, whether between ourselves or in collaboration with our clients. But this had a slightly different twist. We gathered a group of representatives, from multiple different businesses, across multiple different verticals, in a room to experiment with how well different approaches to ideation perform. Every attendee left with an understanding of the different approaches, and insight into which approach they felt they could apply most effectively with their own teams.

For those of you that couldn’t make the session, we want to share some of those ideas so that you too can run ideation sessions that generate impactful ideas.

 

But first, why does the ideation process matter?

Ideation injects creativity into a data-driven process

Data is incredibly important in driving all of the decisions we make in experimentation. Data is great at telling us where the problems are, but it isn’t good at telling us how to solve these problems. This is where creativity plays a key role in experimentation. Ideation is our opportunity to inject creativity into what we do, to explore new concepts and experiment with potential solutions as we hone in on the optimal solution for each problem.

Our time is limited

For many of us, there’s barely enough hours in the day to get through all our emails, let alone spend hours in unproductive brainstorming. We need to optimise how we spend our time in ideation to get a good return on our time investment. The ideation process needs to reliably generate high quality ideas that can immediately be tested through experimentation.

More people doesn’t always mean better ideas

We’ve been taught that collaboration should generate more and better ideas, but this is only true when people can contribute effectively. When more people are in the room, there is often a tendency for people to become protective of their own ideas, as opposed to sharing, discussing and letting their ideas evolve through the input of the rest of the group. Everyone present in an ideation session should be able to, and also expected to contribute effectively.

Subtle differences in execution can have a big impact on results and build time

The difference between an execution of an experiment that wins versus an execution that loses can be extremely small. Two ideas that seem similar can often perform very differently when exposed to real users. One way of testing an idea might require a lot more effort to create than another. Ideation plays a key role in both defining and refining ideas.

Top companies attended to take away some actionable insights from the ideation session
Top companies attended to take away some actionable insights from the ideation session

Our top 4 tips for effective ideation, no matter your approach

1. Separate the hypothesis and the execution 

Ideas that come up in an ideation session come in all shapes and sizes. Some will come to you fully formed with a well-defined hypothesis and a sketch of the execution of the experiment. Others will be less-formed, and need to be refined into a clear hypothesis first before ideating around the best execution of that idea. When an idea is raised, identify whether it’s a hypothesis or a specific execution. For hypotheses, you can then ideate on solutions. For executions you can step back and ideate around the underlying hypothesis of the idea.

2. Break ideas up into sequences of experiments – starting with the minimum viable experiment

One way of doing this is to think about how the idea might fit into a sequence of experiments. Think about the idea you want to explore. Is it the ‘minimum viable experiment’? Or is it a more formed exploration of a specific solution in this area? Think about how you might take your idea and iterate on it across a range of experiments, continuing to improve, and reap results of the idea you have produced to finally reach that ‘perfect’ version.

3. Is this idea iterative, innovative or disruptive?

When you reach that stage of an ideation session where the ideas start to dry up, a useful exercise is to group your ideas into 3 types. Which are iterative ideas – tweaks, optimisations or designs? Which are innovative ideas – new experiences, journeys or usability? And which are groundbreaking, disruptive ideas that will affect product, pricing, or even the company proposition? Try and make sure you have a good number of each category.

4. Have a plan to weed out your bad ideas

Remember back at school, when you were told to put your hand up and express your ideas, whatever they were? There’s no such thing as a bad idea, right? Wrong.

Contrary to common belief, there is such thing as a bad idea. We would encourage you to have a completely open system for idea creation. Allow people to come up with whatever ideas they like. But, also have a system for critique and review. Some ideas will simply not be good, others will be good, but won’t be possible.

There has to be a good mix of realistic, and ambitious ideas. The last thing anybody wants to do is waste their precious time talking about ideas that are simply ‘dead in the water’. One approach we like for this is using The Disney Method.

 

The 3 approaches to ideation we tested

We’re going to share with you the three approaches that we explored in our ideation workshop.

Unstructured Ideation

Quite often when we talk to people and ask how they ideate, the answer is “Well, we get everyone in a room and talk about some ideas”. This is the more common type of ideation session, and by no means the approach that we would recommend. But, it does provide a useful baseline, and if you apply all of our tips above it can, in the right circumstances, still be effective.

That said, it’s pretty much a free-for-all. Everyone shouts their ideas, there is very little focus, it’s unstructured. For this session, we gave attendees a vague goal of increasing conversion rate and a specific page of a website to improve – to mimic the setup of one of these sessions in a conversion optimisation context.

Pros: 

+ Anywhere, anytime

+ Anyone can do it

Cons:

– Anyone can do it (the input expected from each attendee isn’t clear)

– The ideas generated tend to be unrelated and broad

– Small number of high-quality ideas

Structured Ideation

In most cases, when we talk about ideation we’re talking about the creation of ideas for AB tests and experiments. What many people fail to remember, is that experiments are just the end-product of a larger strategic process. At Conversion, we build our strategy around our experimentation framework.

Most unstructured ideation sessions tend to be around a loosely defined goal and perhaps a KPI to be improved. However, in order to conduct an effective ideation session we need more structure and focus.

Only after you have defined your Goal and KPIs, then used data to understand and define your Audiences, Areas and Levers, should you start to ideate for experiment concepts. Agree the specific audience, area and lever that you’re going to ideate on and make sure everyone knows this, and has seen any relevant data and research before they attend. The session will then be more focused around solving a specific problem, and structured.

For this session we gave attendees a completed experimentation framework and defined the Goal, KPIs, Audience, Area and Lever we wanted them to ideate on.

Pros:

+ Customer-focused

+ Impactful concepts

Cons:

– Needs upfront research

– Takes longer to get started

Crazy 8s

This is one of our favourite and most enjoyable methods of ideation. It forces every member of the group to produce 8 ideas, in 8 minutes. We’ve adapted the concept from one originally adopted by Google.

Our adaptation of the original Crazy 8s system is to apply it to a structured ideation setup. So again we have defined our Goal, KPIs, Audience, Area and Lever. Then we use the Crazy 8s ideation process on that specific lever to generate a large number of ambitious ideas in a short space of time. Rather than one person generating 8 ideas on one lever, we often rotate the paper so that you have 1 minute to add a new idea on a lever that nobody else has come up with yet. In this way you can cover multiple levers in one session if you’re ambitious.

The purpose of using Crazy 8s is to force everyone in the ideation session to contribute, but also to stretch all of the attendees to contribute more ambitious, creative ideas then they might generate without the added time pressure. It also encourages people to draw ideas to save time, which can bring out new ideas as people get more visual.

Pros:

+ Large number of ideas

+ Includes all the structured ideation benefits

Cons:

– Needs an introduction

– Less collaborative

 

Our conclusions

Overall, we thought the workshop was a great success. We had some great feedback from those involved and some brilliant ideas for our attendees to take away.

The key takeaway was that structure is crucial for effective ideation. And by that we don’t mean the minute-by-minute structure of the session itself, we mean the structure, focus and setup of what you’re ideating about. A structured approach not only generates more ideas, but generates crucially more impactful, creative and ambitious ideas. As the host of the session you will walk away with both the quantity and quality of ideas you need to design your experiments.

Ideation is critical to experimentation. In order to create an effective experimentation roadmap, you must engage in effective ideation. Following just a few of these techniques, will have you well on your way. But of course, if you’d like to know more, do get in touch.

 

If you’d like to attend future events, keep an eye on our events page.

Conversion.com hosts… ‘Experimentation Maturity: What advanced testing teams do differently’

We are very excited about the release of our Ecommerce Performance Report for 2018 in partnership with Econsultancy. The report covers various concepts in-depth ranging from the growth of the ecommerce market to the future of experimentation however, one thing got us talking at HQ – experimentation maturity.

Out of the 400 ecommerce professionals surveyed, 50% stated that they perceived the value of experimentation to be high/very high however, only 14% stated that their business recognised it as a strategic priority. The disparity between value and strategic prioritisation got us thinking about the roadblocks to experimentation maturity and how we can overcome them.

Kyle Hearnshaw, Head of Conversion Strategy, Stephen Pavlovich, our CEO and James Gray, Senior Optimisation Manager at Just Eat, led our second independent event where we brought practitioners from the industry together to discuss five key themes around experimentation maturity:

  1. What defines a mature approach to experimentation and conversion optimisation?
  2. How can you measure your growth in maturity over time?
  3. How does your organisation stack up compared to 450+ respondents in our report?
  4. What challenges do we face in developing maturity?
  5. How can you overcome these challenges?

Measuring experimentation

In the past we have seen organisations use the size of the experimentation team, the number of experiments launched or the complexity of experimentation as metrics to measure their experimentation maturity.

At Conversion.com, we believe such metrics should be avoided and that maturity should be measured against quality. This means moving the goalposts so that we are benchmarking against experimentation goals, experimentation strategy and data and technology strategy.

What are you trying to achieve via experimentation? 

Setting goals in which to measure the success of experimentation is pivotal for any organisation. At Conversion.com, we recommend following three steps to develop robust experimentation goals:

  1. List your key business challenges – the most mature experimentation programmes drive the strategic direction of businesses. If you feel a long way off this, don’t panic. Listing business challenges and making sure experiments have a measurable impact against these is a huge step on the journey to maturity.
  2. Set a specific goal for experimentation – individual experiments need specific goals and KPIs. In order to translate business challenges into specific experimentation goals we recommend using a goal tree map.
  3. Plan a roadmap to develop maturity – recognising your business’ position on the maturity scale is important in identifying necessary steps to reach maturity. At Conversion.com, we have created an experimentation maturity assessment so you can easily plot where you sit on the scale; be honest, a realistic benchmark allows you to identify the key steps to take in order to progress.

How do you organise and deliver experimentation? 

Experimentation strategy often gets forgotten about, as a consequence holistic experimentation can be derailed and the link between business strategy and goals can break.

In order to prevent this disconnection we recommend that you:

  1. Set regular points to review strategy – stand back at regular intervals to look at the big picture. Involve stakeholders from across the business and brainstorm strategic priorities.
  2. Organise experiments into projects/themes – Ad hoc tactical experiments can be chaotic therefore, we recommend defining projects that group related research, experiments and iterations. This allows goals and outcomes to be measured at a project level.
  3. 10x your communication – The goal here is to get more people involved in experimentation. Some great examples were shared at our event – Just Eat talked us through their approach to ensuring experimentation strategy is shared and reviewed across areas of the business with their experimentation forum. Within this forum every product manager pitches their experiments, detailing what they’d like to test and how they plan to measure the results. We think this is an excellent way to create open communication across a company.

How do you measure the impact of experimentation?

Evaluating the impact of experimentation is pivotal in order to determine the success of testing. But what measurements are important?

  1. Define standards on experiment data – at Conversion.com, we talk about ‘North Star’ metrics. Experiments that have an impact on these metrics should get noticed by senior stakeholders. This tactic resonated with those at our maturity event – organisations voiced that they struggled to get executive buy-in without proving value through results.
  2. Strive for a single customer view – a single customer view is an aggregated, uniform and comprehensive representation of customer data and behaviour. Going beyond single conversion metrics is a huge leap in the path to maturity however, it is not easily achieved. We recommend integrating testing tools with business intelligence tools in order to gather data such as lifetime value.
  3. Build in segmentation as early as possible – no matter where you are on the maturity scale we highly recommend identifying key audiences in order to lay the foundations for personalisation. With this in place, you can report on audience behaviours within experiments and uncover greater insights from your experiments.

So, what next? 

Identify your organisation’s maturity level using our maturity model. Whether you are just getting started or are an advanced team, there are actions you can take to reach the highest levels of maturity.

We thoroughly enjoyed hosting our second independent event. Insightful discussions were had across a variety of organisations and we are proud to have been able to offer advice to assist organisations on progressing towards their own experimentation maturity.

If you’d like to attend future events, keep an eye on our events page