Matt:
How do you actually realize value from running experiments in your business?

Traditionally, people think of experimentation as conversion rate optimization — improving the performance of your site. And that’s important. But as experimentation programs mature, they start to unlock a different kind of value.

It becomes a tool to help the company make better decisions.

I was just talking to James about this, using the example of a major manufacturer. They asked: “We have 10 different brands. When customers shop at retailers, they can see all our products across brands in one place. Should we replicate that experience on our own site?”

That’s not just optimization — that’s a strategic decision a product manager or executive has to make.

So, could we think about experimentation as a way to help prioritize and inform those kinds of decisions?

Stephen:
Exactly. I like to think of it this way: experimentation shouldn’t just validate what you were going to do anyway. It’s not about patting yourself on the back for a good idea.

Instead, it should be used as a way to manage risk.

That’s when experimentation is most valuable — testing the ideas that make you nervous.

Imagine you’re playing a video game. You hit a checkpoint, try something bold, and if it doesn’t work, you restart from the checkpoint. Experimentation gives businesses a similar safety net. You get to test bold ideas without fully committing to them.

For many companies, that’s where the real competitive advantage comes in. If your competitors are sticking to safe, middle-of-the-road ideas, there’s a huge opportunity in testing the things others are too nervous to try.

Matt:
So give me an example of that.

Stephen:
Sure. Think about pricing.

• What if we introduced two new pricing tiers — or eliminated them all and just offered one?

• What if we switched from monthly billing to annual plans only?

• What if we completely reimagined our commercial model?

Those are big, bold ideas. Most companies would never try them outright. But with experimentation, you can run them in a controlled, low-risk way.

And that’s the real value: it helps you make decisions you’d otherwise be too nervous to make — and potentially gain a competitive edge.

Matt:
So how do you prioritize between these safe optimizations and these bold, high-risk ideas?

Stephen:
We use a risk-based portfolio approach, similar to managing investments.

Most of your experiments should be safe or medium-risk — they won’t drastically move the needle, but they’re steady bets.

But you also want a portion of high-risk, high-reward experiments in the mix. That’s where the big breakthroughs come from.

And you can adjust your portfolio based on context. For example:

• If it’s a slower sales period, you might take bigger risks.

• If you’re in an emerging, disruptive industry, you might lean toward bold experiments to get ahead.

It’s about being deliberate in how you balance those safe bets with those high-risk plays.

Matt:
That’s fascinating. It also makes me think we could do another episode just on experiment strategy by company type and industry — because one size definitely doesn’t fit all.

Stephen:
Exactly. Every company needs its own strategy based on its goals, industry, and risk tolerance.

Matt:
Brilliant. Thanks again, Stephen — always great insights.

Stephen:
My pleasure.