Why your experimentation program needs a risk profile

Risk can be a competitive advantage in your experimentation program.

If you’re taking more calculated risks than your competitors, you’re going to get better results.

But to do that, you need to understand your risk profile.

In this post, we’ll look at how to define risk in your experimentation program – as well as three techniques to create better high-risk experiments.

“In a world that is changing really quickly, the only strategy that is guaranteed to fail is not taking risks.”

Mark Zuckerberg

Facebook have always been open to taking risks.

When he was 22, Mark Zuckerberg turned down billion-dollar offers for the company. Instead, over the next ten years, he spent billions of dollars himself: acquiring companies like Instagram and WhatsApp, and making other long-forgetting investments and product launches that didn’t pay off.

The priority is to manage risk – and experimentation can be an effective strategy to do so.

Now some stakeholders will think that any experiment – no matter how small – is a huge risk. But for most A/B tests, your risk is limited. There’s the cost of building the test and occasionally a potential drop in performance during the test. But normally that’s it.

In fact, there’s actually much more risk in making changes to the website without testing them.

Your experimentation program already has a risk profile.

Every experiment you run is low risk, high risk, or somewhere in between. And if you’re not consciously managing that balance, you’re probably not getting the full benefit of experimentation.

Experiments can be low risk. You might have run similar tests in the past, and be pretty confident that this one will work. (Or at least confident it won’t break anything.)

Low risk: If showing “most popular” sleeve lengths was successful (left), then expanding the sizing options out (right) is likely to be low risk.

They can be medium risk. You might be trying out a completely new untested hypothesis. It could work – or you might have wasted time building the test, and lost money running it.

Medium risk: What would happen if we added a donation target to Unicef’s landing page, and showed how close we were to achieving it?

Or the experiments can be high risk. This is when you test disruptive ideas. Experiments like this are high risk because you’re risking the cost of building it and the potential loss of money while the experiment is live.

But it goes further – there’s also a risk outside of the experiment. It might affect the audience in the experiment long after you stopped it – or it might have implications on the brand as a whole.

High risk: What would happen if we changed Wistia’s SaaS pricing model from feature-based to volume-based?

To summarise:

Low-risk experiments are typically iterative – you’re building on an already-proven concept. Their role is to exploit: you’ve validated a lever and are now looking to maximise its impact across the customer journey. The only potential loss is the cost of building the experiment (just because it worked once, doesn’t mean it’ll work again).

Medium-risk experiments are typically innovative – you’re testing out new concepts (but not necessarily radical ones). Their role is to explore: you want to understand what drives customer behavior, and an experiment will inform that understanding. As before, the potential loss is the cost of building the experiment – but you may also lose money running the experiment, if it lowers performance.

High-risk experiments are disruptive – not only are you testing out something new, there’s a chance that it could fail miserably. These are the concepts that your competitors are probably too nervous to test – but they could deliver you a significant competitive advantage if they work.

Their role is to expand – to widen your approach by testing radically different ideas. But the risk is greater too. There’s potential for non-controlled impact – essentially, where the damage doesn’t stop when the A/B test stops.

Take the screenshot above from Wistia’s pricing page. Testing a new pricing structure is a high-risk experiment: it could significantly increase revenue, or it could lower it. And potentially it could affect customers who aren’t in the experiment, it could be reported on social media or wider, and so on.

But often these high-risk experiments come with the highest reward. These are the ones that help you move beyond the local maxima.

High-risk experiments will help you jump from the local maxima to the global maxima.

Look at the experiments you’ve run in the last 6 or 12 months, as well as your backlog of upcoming experiments. Then rate each as low, medium or high risk.

Of course, the definition of risk in your organization will be different to mine. So come up with a simple format that works for you.

If you like, you can try a series of questions like this:

• What type of change are you making? eg UI, functionality, pricing, product.
• Have you tested a similar hypothesis before?
• If you have, was it successful?
• What’s the cost needed to build the experiment?
• What percentage of online revenue does the experiment affect?
• Might it change the behavior of users in the experiment even after it’s stopped?
• Might it change the behavior of users not in the experiment?

We’ve put this in a simple spreadsheet. You can answer all the questions and get a risk score straight away. Of course, you’ll want to adapt the questions and variables and scoring before you start. This is just an example:

Download and adapt this Google Sheet

Or if you want an even simpler alternative, just ask yourself this question about each experiment:

“If I couldn’t run an A/B test, would I still make this change?”

If you’d still make the change, it’s almost certainly low or medium risk. If you wouldn’t, it’s probably high risk.

If we only test changes we’d make anyway, we’re wasting the opportunity of experimentation.

This is one of the most common mistakes people make in experimentation. They only run tests on changes that they’d make anyway.

It starts with an idea:

“This seems like a good idea. Let’s test it and see just how right I am.”

Now there’s a good reason to test these changes. You might be wrong. Or some audience segments may respond differently. And if it is successful, it’s good to know the size of the impact – not just whether it’s positive or negative. This insight will help you come up with new hypotheses and prioritize your roadmap.

But it’s just as important to test changes that make you nervous. Disruptive experiments allow you to make bigger bets.

“This experiment might crash and burn, but if it works…”

If you’re only testing best practice or patterns you see on competitor websites, you’re not going to be getting a competitive advantage. You’re going to be limiting yourself to the local maxima.

Experimentation allows us to test anything we want – and to limit the fallout. It derisks innovation.

If you used the Google Sheet above, it’ll show you what your risk profile looks like visually:

In this example, you’ll see that most experiments are blue (medium risk), with an equal balance of low- and high-risk experiments.

There’s no perfect answer for what your risk profile should be. Ideally you’d have a balance of all three – and it should change over time.

So right now, your risk profile might look like this:

You’ve got an even balance of innovative and iterative experiments, with occasional radical experiments included to allow for greater leaps forward.

But if you’re in peak season, it might look like this:

You increase the iterative experiments to reduce the risk. Because iterative experiments have a higher win rate, you’re going to have a safer program during peak season. That means you increase revenue without risking revenue at peak. And you might hold back on radical experiments altogether.

But if you’re just starting your experimentation program then it might look like this:

You have an even balance across all three. You don’t invest too heavily in iterations, since you haven’t tested too much yet. And you balance innovative and radical ideas to get quick feedback as you develop your product and marketing strategy. (Of course, having this many disruptive experiments is dependent on having the right culture.)

By this stage, it’s probably clear that we need to be running more disruptive experiments. This will give us the most meaningful advantage over the competition.

But how do we come up with the ideas?

It can be challenging to think creatively. We have a tendency to think incrementally. To look at what we have already, and see how we can make it a little better. It’s hard to throw it out, start over and come up with something that may be better.

So here are three exercises to try:

Invalidating lore

This is a good place to start.

“Lore” means the anecdotal knowledge or opinions within a company, which have never been tested. The things you do because “that’s the way they’ve always been done”.

Chamath Palihapitiya started Facebook’s growth team. You might have seen him on the news recently saying that the US shouldn’t bail out hedge funds and billionaires during COVID-19.

He attributed the growth of Facebook to their constant focus on “invalidating lore”:

“One of the most important things that we did was just invalidate all of the lore… All we did was disprove all of the random anecdotal nonsense that filtered around the company.”
Chamath Palihapitiya

Remember, this was a company that was only a few years old. But that’s one of the reasons Facebook has been so successful – everything can be challenged, and data > opinion.

That is perfect for us – we deal in data, so we should be able to challenge assumptions and show evidence.

Domino’s Pizza – never afraid to innovate

Take Domino’s Pizza. They’ve been making pizzas for 60 years – there’s a lot of knowledge and experience in the business, but that brings with it plenty of “lore”: a lot of assumptions and “that’s the way it’s always been done”.

But luckily Domino’s is an innovative organization.

So when companies like Deliveroo and Just Eat and Uber Eats started growing, Domino’s had to respond. These new competitors had deep pockets, and were growing aggressively. What’s more, they were charging customers for delivery – while with Domino’s, customers took free delivery for granted.

This put Domino’s in an awkward position. They could carry on as before – offering free delivery, because that’s what they’d always done and that’s what customers expected.

Or they could test it.

So they chose one geographic market to test it in. Then they ran an A/B test that added surge pricing to visitors in the treatment. That way, they could see the effect of making this change. Would customers abandon their order? Would they be less likely to come back in the future? Or has it become accepted that you pay for food delivery now?

What’s important here isn’t the result of the experiment – it’s that you seek out the lore in your business, and test it. Forget what you’ve done before or what you thought you knew, and focus on what it could be.

Divergent thinking

Next is divergent thinking. This is where you answer a question again and again and again… until you start coming up with weird and wonderful answers. Then you go back, and build on these ideas.

Name as many uses for a brick as you can.

You’ve probably heard of interview questions like, “Name as many uses for a brick as you can.”

So you might start by talking about building a house or a wall. Then you might think about its attributes. Bricks are heavy, so you could use one as a paper weight or door-stop. Or you could use it to break a window, or take the wheels from a car, and so on.

This is the approach that Airbnb founder Brian Chesky used to design their “11-star framework”.

Chesky and the other founders wanted to create the perfect experience. So they started to brainstorm the equivalent of a hotel’s 5-star experience, and extrapolated from there.

Do listen to Chesky talking about this himself – it’s hard to do it justice second-hand. [Masters of Scale 10:36–13:23]

You almost have to design the extreme to come backwards.
Brian Chesky

Like Chesky says, you design the extreme and then come backwards. That way you create something that’s significantly further ahead of where you’re at today.

You don’t take one step forward from where you are today. You go to where the ideal becomes impossible and then take one step back.

We need to shift our mindset from incremental growth to exponential growth. It sounds obvious, but the following is a mistake I make all the time…

You’re building an experiment backlog, and you start by looking at what’s already there, what it looks like, how it works…

But often, you anchor your ideas to what’s there already, making small changes rather than throwing things out and starting again.

And that’s totally fine most of the time – especially if you’re deliberately working on low-risk experiments.

But if you want to come up with radical ideas, you need to think differently. So ask yourself:

What would increase the conversion rate 2x (not 2%)?

Let’s take an example…

You’re optimizing a SaaS website and you want to increase sign-ups. You could…

• Improve the homepage based on user feedback (iterative)
• Optimise the form based on best practice (iterative)
• Emphasise Google or Facebook sign up options (innovative)
• Change from a single-page form to a chatbot style Q&A approach (innovative)

Or could you remove the registration step altogether?

This is what Posterous did, ten years ago. They wanted to get more people to host content on their blogging platform.

But rather than forcing people to sign up, they just let people upload the content by sending an email. Then they’d create an account automatically:

This is one of the hardest things to do in experimentation – to look at a problem differently and not just test changes you’d make anyway.

Good luck!