Matt Wright:
Hello, and welcome to another coffee break episode of The Conversion Podcast, where we provide easy-to-digest and easy-to-apply insights on digital experimentation, conversion optimization, and customer experience,  all in the time it takes to grab a coffee. James, thank you so much for joining today.

James Flory:
Absolutely.

Matt Wright:
I’m excited about this one. I’ve had the good fortune to work closely with you over the last eight years, collaborating, traveling, getting into debates, and having side discussions. You’ve worked across tons of brands and have a wealth of experience, especially in using experimentation and testing not just for optimization, but as a tool to help businesses make decisions.

To tee this up, you’ve talked about a mindset around experimentation, the idea of being “pragmatic vs. purist.” Can you share how you define those approaches and why it matters?

James Flory:
Yeah, for sure. And thanks for the flattering intro, hope no one can see my head getting bigger on camera.

When it comes to pragmatic versus purist experimentation, I’ve seen both sides over the past ten years in the industry. The purist approach is where people get deep into the details, AB test design, statistical significance, bias, and valid experiment setup. It becomes this echo chamber of who’s right or wrong.

But the pragmatic approach is more focused on why we’re running experiments. Most of the time, it’s to answer a business question. A company wants to invest resources or choose a direction, and we’re there to help collect data to support or invalidate those decisions.

So, instead of being stuck in the weeds of “how” to run an experiment perfectly, we need to remember the purpose: enabling decision-making. The stakeholder defines how much certainty they need and how fast they need to move. We’re not in healthcare, we’re not saving lives here. The goal is to provide clarity for business decisions.

Matt Wright:
I love that. Can you give an example of a business question you’ve helped answer using that pragmatic approach?

James Flory:
Sure. One example was a large manufacturing company with a direct-to-consumer business. They had multiple brands under one corporate umbrella. Each brand had its own website, but they were also competing with big-box retailers who sold their brands alongside competitors.

Their executives came to us and asked: “Are we doing ourselves a disservice by not cross-promoting our other brands on our individual brand websites?”

That’s a big question. Moving to a single unified site would be a massive investment, and what if it doesn’t work?

So we used a combination of methods to investigate. First, we looked at case studies. Gap, for example, has a central nav linking out to all its brands, but each one still has its own website. Same with Logitech. That gave us something we could actually test.

We ran experiments by adding cross-brand navigation links and observed: did users notice? Click? Shop other brands? Did it increase revenue or cannibalize it?

We also conducted qualitative research, including interviews and usability tests, to determine whether customers actually wanted this kind of experience. That told us the why behind the behaviors.

So instead of chasing a gold-standard, statistically perfect AB test, we gathered a full picture with multiple data points. That was enough for the business to confidently make a decision, even without 95% statistical certainty.

Matt Wright:
That’s such a great example. I think people often get stuck thinking they need total certainty. But in reality, testing humbles us. Things we expect to work often don’t.

So with so many potential methods and approaches available, how do you decide which one to use and how much data you actually need?

James Flory:
It depends, classic frustrating answer, right? But it’s true. There are a few key considerations:

1. How fast do you need to move? If speed is critical, you might not have time for high-certainty, slow methods.

2. How much risk is there if you’re wrong? Some decisions are low-stakes (like choosing an image), others are high-impact.

3. What kind of data will answer your question? If it’s about behavior, use quantitative methods. If it’s about desires or motivations, start with qualitative research.

Sometimes it’s a mix of both. Use whatever gets you to enough clarity to move forward. Don’t fall into the trap of perfectionism or thinking only statistically significant results matter.

Matt Wright:
Totally. Are there any foundational tools or processes someone should have in place to adopt this approach?

James Flory:
Honestly, just the will to do it. Qualitative research doesn’t need much, talk to customers in-store or online. Use surveys, free tools, or platforms with built-in audiences.

If you’re going quantitative, you’ll need a testing tool, a clear hypothesis, and the ability to run an experiment. But even small, underpowered tests can be valuable if they help you move forward.

Don’t get discouraged if you’re not perfect at this right away. Just start collecting data and using it to make smarter decisions. That’s progress.

Matt Wright:
Great advice. Last question, how do you translate these insights to different stakeholders? You’re great at this, and I know many people struggle with it.

James Flory:
Thanks. I think every experimentation program needs a “killer test,” that one surprising, impactful result that grabs everyone’s attention.

Find the question everyone’s debating or the decision they wish they’d tested before. Run an experiment that answers that.

A perfect example is what we did with Whirlpool. One of their products already had a 90-day money-back guarantee, but it wasn’t visible on the site. The product managers thought, “Why not add it? Everyone loves a guarantee.”

So we tested it. And when we put it on the site, sales tanked. Completely counterintuitive. That test became the program’s “killer app.” It got everyone’s attention and made them think: “Wait, what else should we be testing?”

So if you want stakeholders to care, go find your killer test, the thing that makes people say, “Wow. I didn’t see that coming.”

Matt Wright:
I love that. Thanks again, James. I know you’re super busy, so I really appreciate you taking the time to share all this today.

If someone wants to connect with you or follow up, what’s the best way?

James Flory:
Connect with me on LinkedIn, send me an email, or even just comment something controversial on one of my posts to get me riled up. Whatever works!

Matt Wright:
That might be the best method! Thanks again, James, I appreciate it.