Hello, and welcome to another coffee break episode on The Conversion Podcast, where we provide easy-to-digest and easy-to-apply insights on digital experimentation, conversion optimization, and customer experience in the time it takes to grab a coffee. Thanks again for joining, Steven. The one that we wanted to tee up was the idea of risk profiles in the context of experimentation.

So with that, maybe just dive right in. Love for you just maybe give a little bit of background of, like, what is a risk profile really in the context of when you’re running online experiments? How should it be used? Why is it important? And, can dive in from there.

Steven: Yeah. Absolutely. So we think about risk profiles in the same way that you might do with an investment portfolio. So if someone were investing, they might have some low-risk options, some medium, some high risk. And the low-risk ones are the kind of the more steady investments, I suppose, that would be delivering fairly consistent growth. Whereas the high risk at the opposite end, it might do amazing things or it might turn to dust.

And we want to think about that in the same way with experimentation because the ability to A/B test gives us effectively a safety net. So we think of it as a way to manage risk. When you take that concept and apply it to the different ideas, the different hypotheses that you can run, it really unlocks a whole new way of thinking that can encourage an organization to be a lot more aggressive in their thinking. You can unlock a lot of competitive advantage as well.

Matt: Oh, yeah. Very cool. I think if I think back myself personally, I feel like I heard you talk about this maybe six, seven years ago for the first time.

Steven: Yep.

Matt: Yeah. I’m curious if you take me all the way back to there or even probably before. What was really the challenge that caused you to really start championing the idea that every program should have a risk profile?

Steven: Yeah. For sure. So, I guess in the early days of experimentation — or CRO as people tended to call it back then — a lot of people were thinking about experimentation in terms of running fairly basic tests like, you know, let’s tweak this text or let’s, you know, the classic change the color of the button, or even once like, what happens if we add an outline to the button or add a shadow effect?

And we’re thinking like this is not what we’re doing at Conversion. Like, we think about experimentation in a different way. And sure, those kinds of tests sometimes can have an effect, but we wanted to encourage people to think more creatively about experimentation and to use A/B testing as a way to test some of their biggest and boldest ideas. So we started to espouse this idea of risk profiles in experimentation that, yes, those smaller, more incremental iterative ideas have a place, but actually, we should be testing far bigger ideas. The things that we would be too nervous to just roll out, just make the change and hope for the best.

But because we can A/B test it, we can limit the fallout to an extent. And if it works, then we’re going to be testing the things that our competitors are scared of trying. And so this is how we can get that competitive advantage.

Matt: Okay. Nice. Maybe to make it quite tangible, especially if someone’s listening right now — what are ways to think about different categories of risks in an experiment?

Steven: For sure. So we have a fairly simplistic view. We talk about low-, medium-, and high-risk experiments.

• The low-risk ones are more iterative in approach. You might be building on a concept that you’ve already tested and validated previously, but you want to push it a little bit further. The only real risk is if you run the experiment and it doesn’t work, it’s probably not going to cause too much disruption. So really the only thing you’re risking is the time that it takes to build and run the experiment itself.

• Medium-risk experiments tend to be a little bit more innovative. If low risk is exploiting something that you know works, medium risk is more exploratory. You’re trying out new ideas, and you’re risking the time it takes to build the experiment. But there’s a potential that it might have a negative effect, and there’s a small — but worth noting — chance that you actually reduce sales or conversions during the test.

• Finally, high-risk experiments. If the first two were iterative and innovative, then high risk would be the disruptive ones. These are ideas that challenge the existing way of thinking.

As a quick example, you might be testing out a completely new commercial model, or selling a new product category, or — like in one example we discussed — removing a free trial that’s always been in place. These are big, bold steps you might be nervous to take without an A/B test.

The risks: the cost of building, the chance of lowering sales, and potentially a bigger fallout beyond the experiment (like reputational risk). But that doesn’t mean you shouldn’t do it — just that you need to plan ahead.

Matt: Very cool. I think this is a powerful way that forces you to think about experiments in aggregate rather than individually. Have you seen people actually apply the risk profile approach and change the overall direction of their program?

Steven: Yeah. Absolutely. Often when we work with clients, they’ve already been experimenting, but usually in the low- to medium-risk range. When we introduce risk profiles, we don’t immediately say “a third of experiments must be high risk.” Instead, we build trust in the process and gradually nudge toward bigger bets.

We’ll ask questions like: If you could test anything, what would you try? What are your competitors too nervous to do? What’s accepted as true in your industry that no one’s ever questioned?

A concrete example: Domino’s Pizza in the UK. They’d always offered free delivery. Meanwhile, platforms like Uber Eats or Deliveroo were charging service and delivery fees. So Domino’s asked: maybe customers expect to pay for delivery now, and we’re missing out. We tested adding a £3–4 delivery fee. That’s the perfect high-risk experiment — bold, backed by evidence, risky, but potentially transformational.

Another: a SaaS company we worked with removed their free trial. Conversion rates dropped, but every new signup was paying, and profit rose 20% overnight.

Another: a UK pharmacy brand. We knew shipping mattered to customers, so we tested same-day pickup with a “painted door” (fake option) before building the feature. Customers jumped at it, which justified the tech investment.

Matt: That’s a great list. If someone wants to build their own risk profile, how should they start?

Steven: First, just be aware of the concept. Then, label experiments in your backlog: low, medium, or high risk. Color-code them. Step back: are you 80% low, 20% medium, 0% high? Is that balance right for your company, your stage, your season?

Risk appetite changes. During peak times (Black Friday, holidays), dial down risk. During quieter months, take bigger swings.

It’s also a great planning tool for alignment and discussion. Teams debate: What risks are we taking? What impact size are we aiming for?

To spark high-risk ideas, try exercises like:

• Invalidating lore (L-O-R-E): Challenge assumptions that everyone treats as fact but has never tested.

• Divergent thinking: Ask what a 5-star experience is. Then 6-star. Then 7, 8, 9… Push into the ridiculous, then scale back.

• Think 2x, not 2%: Instead of incremental tweaks, imagine only testing ideas that could double conversion.

Matt: Love that. But of course, sometimes incremental wins add up, and sometimes you need big bets. Netflix, for example, had moments where they only went big to get off the ground, then later optimized relentlessly.

Steven: Exactly. The key is to use experimentation as your safety net, so you can take bold bets without gambling blindly.

Matt: Before we close, what’s your actionable advice for someone who wants to apply this right now?

Steven: Two things:

1. Bottom-up: Review the past year of experiments. Which ones mattered? How could you scale those insights into bigger tests?

2. Top-down: Look at your company strategy. What big initiatives are planned? Design experiments to pre-test those ideas.

That way, experimentation informs both quick wins and long-term strategy.

Matt: Perfect. Any resources for digging deeper?

Steven: Yes — we’ve got a post on our website about risk profiles. Google “Conversion risk profile” and you’ll find it. Still highly relevant.

Matt: Fantastic. Thanks so much, Steven.

Steven: Thanks, Matt.

Matt: And thanks to everyone listening to another coffee break episode of The Conversion Podcast. More to come soon. Have a great day.