Ep 2: How mature is your experimentation program, really?

The Conversion Podcast | Episode 2 | Maturity in experimentation

Matt welcomes Steph, a Principal Consultant at Conversion, to unpack a new and evolved perspective on experimentation program maturity. With years of hands-on experience guiding companies through the realities of scaling A/B testing, Steph brings a refreshingly practical view of what maturity actually looks like and how to get there.

While many organizations track success through test volume and velocity, Steph argues that those metrics alone don’t tell the full story. Maturity isn’t just about speed; it’s about clarity, alignment, and making experimentation a strategic function of the business.

Drawing on a new 5-stage maturity framework, Steph explains how teams can benchmark their current state across four key dimensions: goals, delivery, strategy, and tooling. She outlines how companies often evolve unevenly, with some parts of the program advancing faster than others, and why that’s normal.

Matt and Steph also dive into the challenges that hold programs back, from over-engineering tests at the wrong maturity level to under-investing in velocity tracking. They explore what it really takes to move from a siloed testing function to a company-wide experimentation culture.

Top takeaways include:

Moving from reactive testing to a fully optimized experimentation program
Why maturity isn’t linear and how to navigate uneven growth
The importance of “value” in testing (beyond revenue alone)
Why clear KPIs and lightweight research can unlock momentum
Practical steps to identify constraints and unlock the next stage of program maturity

Episode Summary

A new 5-stage experimentation maturity model to replace outdated frameworks
How to evaluate your program across goals, process, culture, and tooling
Why teams often overcomplicate testing before they’re ready
The difference between speed, scale, and meaningful value in experimentation
Practical steps to benchmark your maturity and move to the next stage

Read transcript

Episode Transcript

Coffee Break Episode: Rethinking Experimentation Program Maturity

Guests: Matt (Host), Steph (Senior Consultant)
Duration: ~17 minutes
Topic: Evolving perspectives on maturity models in digital experimentation

[00:00:00] Matt
Could I say something controversial? I think sometimes people are trying to do testing that’s too sophisticated for their level of maturity, like they’re running before they can walk.

[00:00:13] Matt
Hey everyone, and welcome to another Coffee Break episode of the Conversion Podcast,
where we bring you quick, digestible insights on experimentation, conversion optimization, and customer experience, all in the time it takes to drink a coffee.

[00:00:28] Matt
Today, I’m joined by Steph, one of our senior consultants here at Conversion. We’re diving into how to think about program maturity and introducing a new framework you can use to level up your experimentation.

[00:00:41] Matt
Welcome to the podcast, Steph, great to have you here.

[00:00:44] Steph
Thanks for having me!

What is a Maturity Model?

[00:00:45] Matt
Let’s start simple, what is a maturity model?

[00:01:00] Steph
To me, it’s a way to benchmark your experimentation program. It helps you see where you are, what good looks like, and how to use experimentation to its full potential.

[00:01:14] Matt
Why does that matter for companies?

[00:01:21] Steph
Well, first, it can help you get investment if you’re trying to grow. But more importantly, it helps you move beyond random A/B testing and instead use experimentation strategically to grow your business.

The Old vs. New Model

[00:01:59] Steph
Historically, we used a model called Packet, which focused on five areas:

Process
Accountability
Culture
Expertise
Technology

[00:02:42] Steph
But over time, we realized Packet didn’t give teams a clear visual roadmap to maturity. So I’ve been working on an extended framework that fills in those gaps.

The New 5-Stage Maturity Model

[00:04:25] Steph
Based on my experience, we see experimentation programs fall into five stages:

Reactive – Ad hoc testing, no strategy, often driven by one passionate individual.
Emerging – Starting to gain internal buy-in, small teams form, but still no formal framework.
Strategic – Experimentation is recognized as a business strategy, frameworks start appearing.
Integrated – Testing culture spreads across teams, with leadership support and cultural buy-in.
Optimized – Think Amazon or Netflix, experimentation is deeply embedded in everything.

[00:07:34] Steph
And you’re not stuck in just one bucket. For example, you might be strategic in your goals but only emerging in your tools and data. That’s totally normal.

The Four Key Areas of Maturity

[00:08:16] Steph
We evaluate maturity across four dimensions:

1. Experiment Goals

[00:08:19] Steph
This is about why you’re experimenting.

Reactive: No clear goal
Emerging: ROI shows up occasionally, but not structured
Strategic: Clear KPIs, aligned to business goals
Integrated: Scaling, and goals go beyond just revenue
Optimized: Rigorous KPIs and insights shared business-wide

2. Delivery & Process

[00:10:02] Steph
How are experiments actually run?

[00:10:30] Matt
Quick tangent, can we define “volume,” “velocity,” and “value”?

[00:10:46] Matt

Volume: Number of tests
Velocity: How fast tests go from idea to conclusion
Value: Trickier, can you explain?

[00:11:02] Steph
At first, value equals revenue. But as you mature, it becomes about learnings too. Like fake-door tests that save months of dev time, that’s value.

3. Strategy & Culture

[00:12:07] Steph
This one’s big. It’s about the belief system and mindset around experimentation. Do people embrace it? Are they challenging assumptions and optimizing the process itself?

4. Data & Tools

[00:12:37] Steph
What tools and data are you using to inform tests?

Are your research tools integrated?
Are you only using WYSIWYG editors or running advanced MVTs?
Are your testing tools limiting or enabling you?

Are You Running Before You Can Walk?

[00:13:17] Matt
Hot take, some people try to do complex testing before they’re ready.

[00:13:29] Steph
Totally agree. You need to earn that sophistication. Volume is good, but only if the quality is there.

How to Use the Model

[00:14:05] Steph
Here’s what I recommend:

Benchmark yourself across those 4 areas
Identify where you are, even if it’s a mix
Set small goals for how to move to the next stage

[00:14:47] Steph
Some examples of progress goals:

Define clear KPIs
Do more continuous research
Expand testing beyond just web (e.g. offline, operational)

Lessons from Experience

[00:16:05] Steph
One big trap, missing velocity stage data.
At Conversion, we measure each step, design, development, QA, UAT, etc. If something gets stuck, we know where. Most teams don’t track this, and it’s a killer for process improvement.

Wrapping Up

[00:17:03] Matt
Big thanks to Steph. She’s published tons of great content, check her out.

[00:17:23] Matt
Best place to reach you?

[00:17:26] Steph
LinkedIn. Just drop me a message, especially if it’s about maturity. I’d love to chat.

[00:17:34] Matt
Thanks again, Steph!