[00:00] Introduction

Matt:
Hello, and welcome to another Coffee Break episode on the Conversion Podcast, where we provide easy-to-digest and easy-to-apply insights on digital experimentation, conversion optimization, and customer experience in the time it takes to grab a coffee.

Thank you very much for making time for this today.

Stephen:
Of course. My pleasure.

---

[00:45] Why Prioritization Matters

Matt:
I want to kick this off with a topic you’ve spoken about often — prioritization. Many people overlook it, but you’ve said it’s one of the most critical levers for improving program performance.

So let’s start there: What is prioritization in experimentation, and why does it matter so much?

Stephen:
I don’t know if it’s controversial, but I think prioritization is definitely overlooked.

The best and worst thing about experimentation is that anyone can come up with ideas to test — from internal teams to even their families. So, there’s never a shortage of ideas.

What you do have a shortage of is:

• Traffic

• People to design, build, and run tests

• Analysts to interpret results

There are always constraints. If you can only run 20 or even 100 experiments in a year, you must be careful about which ones you choose and the order you run them in. That’s where prioritization comes in.

---

[03:00] Traditional Prioritization: Impact vs. Ease

Matt:
Even 100 experiments sounds like a lot to some people, but as you’ve said, trade-offs still exist.

How do teams typically approach prioritization today?

Stephen:
Historically, prioritization comes down to two axes:

• Impact (How big of a change might this make?)

• Ease (How easy is it to design, build, and get approved?)

This creates four quadrants:

• High-impact, high-ease (your dream experiments — do these first)

• High-impact, low-ease

• Low-impact, high-ease

• Low-impact, low-ease

Ease is usually estimated fairly well. But impact is much harder to predict, which is why we run experiments in the first place.

The real issue? Bias. People tend to favor their own ideas over others, making the process subjective and flawed.

---

[06:20] Making Prioritization More Objective

Matt:
I’ll admit, I’m guilty of falling in love with my own ideas. So how do we make prioritization less subjective?

Stephen:
Over the years, teams have created scoring systems to add objectivity.

For example, asking:

• Does this change affect something above the fold?

• Is it noticeable within five seconds?

Each answer contributes to a score.

But two problems arise:

1. People game the system — designing experiments to score well rather than solve real problems.

2. No use of historical data — for example, if a free-shipping test succeeded last month, a follow-up experiment should rank higher, but the static model can’t account for that.

---

[09:15] The Magic Wand Model

Stephen:
If you had a magic wand, the ideal prioritization model would:

• Leverage all prior data — not just experiments from one site, but across industries and past research

• Continuously update as new data comes in

That’s where prioritization truly becomes powerful.

---

[11:00] Lever Framework & Meta-Analysis

Matt:
That’s fascinating — but how do you actually organize and analyze that kind of data?

Stephen:
It’s challenging. Most experiment documentation is free text: hypotheses, summaries, etc. There’s no standard way to categorize them.

Our solution was to tag every experiment using what we call the Lever Framework:

• Industry (retail, travel, financial services, etc.)

• Page type (product page, cart, landing page, etc.)

• Change type (copy, image, functionality)

• Behavioral principle — the underlying hypothesis

The lever is key. It distills a hypothesis down to its principle (e.g., trust → security → payment security).

This creates a taxonomy that allows for meta-analysis — understanding which levers are most effective for specific industries, page types, and timeframes.

---

[15:30] Getting Started with Untapped Experiment Data

Matt:
For listeners who’ve run experiments but haven’t categorized them, where do they start?

Stephen:
I’d suggest:

1. Start with experiments that had statistically significant results.

2. Use manual tagging or AI tools to classify them.

We’ve made our lever framework public and even built a tool — levers.ai — where you describe an experiment and get a lever classification instantly.

---

[18:00] Automating Prioritization – Confidence AI

Matt:
We’ve also talked about the dream of a dynamic model that updates automatically. How close are we to that?

Stephen:
We built that into Confidence AI.

It:

• Pulls in all available data on a client’s experiments

• Leverages broader data from thousands of experiments across industries

Then produces a score (0–100) for each backlog experiment — representing our confidence in its potential success.

It updates dynamically as new tests run and insights are gathered.

This doesn’t replace research or ideation. It supports prioritization. Over time, as it learns, it becomes more accurate — helping teams focus on experiments most likely to drive impact.

But human judgment is still essential. For example, if the top five experiments target the same lever, we wouldn’t run them all at once. We’d diversify, like spreading bets at a roulette table.

---

[22:30] Closing Thoughts

Matt:
That combination of human insight and AI feels like the real unlock.

Stephen:
Absolutely.

Matt:
Well, Stephen, thank you so much for this. We’ll add some links in the notes for anyone who wants to dive deeper into this topic, including levers.ai and Confidence AI.

Stephen:
My pleasure. Thanks for having me.