Matt:
Hey, Tim — part two. Setting the scene for this one and diving right into it. I’ve seen you write that successful experimentation is cultivated, not designed or manufactured. You talk about this a lot in the context of experimentation culture, and especially around where people make mistakes. What does that look like, and how should people think about this in their orgs?

Tim:
To put it in one sentence: if you’re trying to build adoption for experimentation — or any kind of organizational change — you have to ask yourself, are you on the soapbox, or are you driving the bus?

The big difference is sustainability. When you’re on the soapbox, you’re telling people what they’re doing wrong. That’s what most of us default to. We see someone doing something incorrectly and our instinct is to correct them: “Hey, you’re doing this wrong — let me show you the better way.”

The problem is, when you approach people by starting with “you’re doing it wrong,” their natural human response is defensiveness. Even people who say they’re open-minded often push back once they feel corrected or judged. There are a few people who love that kind of feedback, but they’re rare.

When you’re trying to expand a culture — whether that’s experimentation, growth mindset, or anything else — spending your time pointing out mistakes isn’t sustainable. Yes, they might be investing in features before validating them. Yes, they might be running gut-driven experiments. That might all be true. But starting there feels like shaming, and it kills adoption.

This applies well beyond experimentation. Any kind of change effort runs into this.

What I recommend instead is what I call bringing people on the bus ride. There’s a book, Trillion Dollar Coach, about Bill Campbell — a former football coach who became an executive coach for companies like Google and Facebook. One thing he emphasized was helping people adopt change by bringing them along, not telling them what to do.

At Minecraft, when I started working with product teams, they didn’t know when to experiment, how to experiment, or what to do with results. Metrics went up or down, and nothing changed as a result. Very few experiments were implemented.

So I spent three months going through the full product lifecycle with them — actually doing the work with them.

We’d start with discovery. I’d ask:
“What are your research questions?”
“What do you want to learn about users that you don’t know?”

Then I’d execute the research. Afterward, I’d bring the team back together and show them what we learned — the problems users were having, recordings, snapshots, examples. I didn’t jump to solutions. I stopped there.

When you bring real user insights into the conversation, you can literally see it click. People start forming ideas naturally. That’s when I’d schedule a separate session just for solutioning.

I try to isolate three phases: observation, theory, and test. If you jump to theory too early, before fully understanding the problem, ideas get diluted and impulsive.

When we did move into solutioning, I’d anchor everything to their OKRs. Otherwise, it becomes “What if we change the button color?” Instead, we focused on what mattered most to them.

From there, we’d prioritize ideas, run tests, analyze results together, discuss theories, make recommendations, and then they’d implement.

After enough repetitions — about three months — something changed. One PM told me, “I will never launch another feature without an A/B test again.”

That’s the difference. The soapbox is a shortcut, but it rarely leads to sustainable adoption. I still catch myself doing it all the time — it’s hard not to.

Matt:
Totally. I think if you spend most of your energy criticizing what people are doing wrong versus objectively presenting data and options, the adoption you get is night and day.

Tim:
Exactly. Even now at Minecraft, my goal is to help all teams adopt experimentation properly. I tell people: “Here’s what we’ve done elsewhere. Here’s the data. Here are potential opportunities. If you’re not interested, that’s totally fine.”

The harder you push, the harder people push back — especially leaders. Leaders believe their judgment and intuition are what they’re paid for. If you say, “You’re doing it wrong,” or “You shouldn’t be making these decisions,” it won’t land well.

But if you say, “Here’s what another team did. Would you be interested in trying this?” — that changes everything. You’re giving them data and letting them decide.

That’s where adoption really happens.

---

Pivoting After the Soapbox

Matt:
If someone’s already been on the soapbox for a long time, can they pivot? What’s a tangible way to shift?

Tim:
First impressions matter. If you’ve already burned trust, it’s an uphill battle. Ron Kohavi talks about this in Trustworthy Online Controlled Experiments.

If you’ve faced pushback, you need to shift from “I have strong opinions” to “I have compelling data.”

You show results. You show cumulative impact. You show how many experiments were run, win rates, lift, revenue impact, retention impact — and you connect that to business goals.

You lean into advocates. You run roadshows. You understand leaders’ goals and tie experimentation outcomes directly to those goals.

The mindset shift is critical:
Not “my goal is to run more tests,”
but “my goal is to help others achieve their goals faster and more effectively.”

That’s how you rebuild trust.

---

AI and Experimentation — The Optimistic View

Matt:
Let’s talk AI. Where are you seeing real impact?

Tim:
I’ll walk through the product lifecycle.

In discovery, AI is a superpower. It can summarize massive datasets incredibly fast.

I ran a survey with 130 people — 680 responses. Pre-AI, that would’ve taken hours or days. With AI, I uploaded the data, asked structured questions, spot-checked outputs, and had insights in minutes.

This works for qualitative and quantitative data. The key is spot-checking. AI can misinterpret context — like “creeper” in Minecraft, or profanity used positively.

That’s where conversation matters. You correct it, guide it, and refine the output.

The biggest value here is speed from data to insight — not idea generation.

I’m cautious about using AI to recommend what to test. It lacks institutional memory — the context that designers, PMs, engineers, and leaders have.

AI accelerates insight. Humans generate solutions.

---

Context, Prompts, and AI Agents

Matt:
Have you found good ways to give AI context consistently?

Tim:
Yes. Either include context explicitly in your first prompt every time, or create AI agents with guardrails.

Agents let you define rules once — context, tone, constraints — and reuse them. That’s especially useful for research and social listening.

---

Design, Development, and Analytics

Tim:
On design, I’m skeptical of AI generating production-ready designs. It’s great for rapid prototyping — visualizing ideas — but not for shipping. It doesn’t account for accessibility or technical constraints.

On development and analytics, there’s huge opportunity. AI can help answer questions like sample size, test duration, and forecasting.

One area I’m especially interested in is predictive modeling — forecasting long-term impact while accounting for seasonality and external factors.

There’s also value in ML-driven analysis: variance reduction, user qualification, and multi-armed bandits — especially for low-risk, top-of-funnel experiments.

---

Community and Learning

Matt:
You opened up time for people to chat with you about experimentation. Why?

Tim:
My manager encouraged me to keep educating and helping others. It creates a forcing function for me to learn.

I’m not an expert — I test, read, apply, and learn. Helping others helps me sharpen my thinking.

Those conversations energize me. I learn from them too. That’s why I did it.

Matt:
If people want to connect?

Tim:
LinkedIn is the best place.