Section 1 — Opening & Introductions

[0:00:00 – 0:01:00]

[0:00:00] Tim Mehta: If you’re not driving measurable growth, you’re essentially taking very big risks and incurring sunk costs on a very frequent basis.

[0:00:13] Matt Wright: Hi everyone.

Very excited once again, to have another great guest on the conversion podcast.

We’ve got Tim Mehta from Minecraft.

Thank you so much for being here.

[0:00:22] Tim Mehta: Tim, yeah, excited to chat.

Cool, man.

[0:00:25] Matt Wright: I think a great way to start all these is maybe to get a little bit of deeper into, like, just your journey leading, obviously, experimentation, or being the lead, we call, like the lead experimentation.

PM, at Minecraft, is that the correct descriptor of it, pretty much, that’s a good start.

Okay, good start.

You can correct me.

Then as we go, maybe just starting with it’s like, what’s your journey to get there?

I think that’s great for people to kind of understand, like, how you’ve actually arrived.

And then if you want to connect it a little bit, I’d love to be like, hey, what does this look like?

What do you actually work on at Minecraft?

So maybe kind of, what’s the lead, and then what are you working on today?

Yeah.

Section 2 — Tim’s Background & Path Into Experimentation

[0:01:00 – 0:06:18]

[0:01:00] Tim Mehta: So my background for the past 10 years plus has very much been in experimentation for marketing, product, any digital property, really.

It started a little bit in the Facebook ads world, where we’re testing different advertisements, and then I got into more of a holistic kind of digital content strategy.

And then from there, web design strategy, working really closely with designers, developers.

And when I got to there, I started to go down this T shaped marketer path.

I was doing SEO Analytics, you know, partial development design.

And then I just kind of fell down this, this rabbit hole of experimentation, and it just made so much sense to me, because it’s it’s so especially coming from a marketing background, it is so enlightening to be able to attribute success to a very particular change you made.

You can ask so many marketers, if you bring up attribution, their heart rate is going to start going up.

They’re going to start feeling anxiety.

With experimentation, it just makes the story so much easier to understand, of what your changes or what new designs you’re putting out there, and how users are behaving because of that very isolated design.

And so from there, I started to work on the agency side, purely focusing on building CRO programs.

I worked with small businesses, medium sized businesses, larger businesses, and I got a really good sense from there of what are the principles of experimentation, especially going from zero to one, what are the pillars that companies or organizations need to truly find success through AB testing, through user research.

From there, I actually got recruited over to microsoft@microsoft.com so this is my first stint of Microsoft where I was focused on the Microsoft Surface product, and that was amazing.

That was a great, great opportunity to see, not necessarily zero to one or even one to 50, but what a program at really high maturity, with resources in every one of those lanes looks like.

And there, I really got to focus on some of the higher level kind of challenges that you get with an experimentation program, which is stakeholder management making sure we’re not just doing tactical little tests here or there, but are we informing business strategy?

Are we helping the product marketing team do their job, so on and so forth?

And so that was a really great experience from there.

And then we had lived on the west coast in Seattle, and moved back to where we live now, in Florida.

And then I started working on at root insurance, just doing CRO for the website.

Got to work really deeply with the data science team right working with an insurance product.

And then I got recruited to Lucid Software, this is a B to B, SAS tool for flowcharts and whiteboarding.

And they asked me to come as the director of growth.

And so this was essentially taking that experimentation background and seeing, how can we spread this practice through our product teams?

And I learned so much there not as much about executing experimentation, but more so, how do we help product teams adopt experimentation and start using it properly.

And that was really fun.

And it was really fun being a manager of people, totally new for me at that point.

And it was, it was just so it was awesome to, like, see your particular PMS, or whoever they may be.

Go from, hey, I don’t understand how to use experiments.

I don’t understand why we should do this to hey, I don’t want to release another feature again without an experiment and then watching them grow so on and so forth.

And then from there, I was approached by Microsoft, but this time from Minecraft, and this was an opportunity where.

I was so excited, because I had never worked at a particular organization that had a product that was not just highly, highly loved by users, but as like had already reached such a high level of success.

I mean, in the 15 years that Minecraft has been around, 15 plus years, it has been the best selling game of all time.

And it was really, really hard to pass up on this opportunity.

And so I joined, starting on the web team as the experimentation PM, and within three months, we found a lot of success.

And then they, they decided we, we should try to get this happening across the rest of the studio, right?

The rest of the organization?

So it started as just the web team, and now my focus is starting to spread out from Hey, instead of just getting the web team running more experiments, you know, making more revenue.

Now let’s see, how can we help the gaming teams?

Right?

The gameplay teams, the marketplace teams, and so I wouldn’t say I’m the experimentation leader.

I mean, on paper, experimentation.

PM, and the focus started with just web, and now is starting to leak out to the rest of the organization.

And now I’m sitting here chatting with you.

Section 3 — Growth vs Experimentation Roles

[0:06:18 – 0:08:49]

[0:06:18] Matt Wright: Well, just my quick curiosity thread or curveball, I’d love to just kind of dive into, just when you said director of growth versus experimentation, I’d be just really curious right off the bat.

Like, how do you think those are linked?

Like, are they complimentary?

Like, what’s different about them?

Like, how do you go about thinking about that from your experience?

[0:06:37] Tim Mehta: So what I noticed, and this is way back on the agency side, was that an experimentation team is essentially operating like a growth product team.

You are going through discovery, doing user research, diving through analytics, looking at market research, talking to users.

You are developing ideas or features for your product.

You are testing those ideas in the real world, and then you’re analyzing them and making the recommendation, should we ship this or not?

So what I described is very in line with what the product development cycle looks like as well, and so, and it was funny, especially at Microsoft, the first time, you know, we were essentially doing a lot of this, this product growth, product work, where our goal was to grow the product, to grow the revenue or retention, or whatever it was we were going after.

So when I got to lucid and started to work as the director of growth, there was a bit of a natural fit, like it was, you know, mostly product work, product focus, maybe 30% experimentation, but it just made so much sense to to bring that experimentation acumen to a growth product team.

If you think about the responsibilities of a growth product, it is activation, retention, monetization, expansion, all things that are very, very that play very nicely with experimentation, right?

These are very clear metrics that you can move by running more experiments, running tests, talking to users, so on and so forth.

And so I think that the connection with growth and experimentation is much, much closer than many people realize, because I mean by nature, the definition of a growth team is to grow the product, grow the success of the product.

If you look at any experimentation team, look at their mission, look at their vision.

It’s essentially the same thing, and you’re leveraging experimentation across every single step of that product development process to get to the end, which for experimentation, for product, everyone’s just trying to make smarter, better decisions that drive more impact.

That’s really the crux of it.

Section 4 — Scale of Experimentation & Minecraft’s Opportunity

[0:08:49 – 0:12:30]

[0:08:49] Matt Wright: Originally, that was very like, limited to just your big tech, gigantic companies, but you can actually scale this essentially, almost in like any growth team today.

So amazing.

Yeah, yeah.

And did you know when you were kind of, essentially, essentially, maybe, like, recruited or in discussions with Minecraft, like, did you know about a lot of this, like, uniqueness of it, like, in terms of the attraction, or were things that surprised you now that you’re actually in the role that you’re like, Oh, I had no idea.

You know,

[0:09:13] Tim Mehta: with Minecraft, it’s hard to ignore, not like, before you even work there, right?

It is, it is, you know, like, I have kids, we walk our kids to school.

There wasn’t a day that I walked them to school where I didn’t see a Minecraft backpack or Minecraft t shirt, and so I was well aware of the success of it, and even coming into it and seeing, right, we have millions of people who visit our properties every week.

It’s just such a great especially in the beginning, it felt like such a great sandbox, I want to say playground, even of opportunities for testing.

It’s, you know, when you have a velocity like that, it’s unlike any other property or, you know, experience that you may have worked on, that I have worked on before, that just had so much traffic.

Like, we still run tests two weeks, just like, you know, that’s really the minimum for any, any experiment, but the speed of which we’re able to start to see and make decisions, I mean, it’s just, it just creates a higher confidence in your outcomes, where, whereas in many places before, right?

Like, even after two weeks, like, do we have the right sample size for this statistically significant outcome to be absolutely trustworthy?

And with Minecraft, we don’t have to ask that question very often.

Like, we see p values that are like, point, 000001, very frequently when we run experiments.

And so I think the only thing that’s really surprised me since coming to Minecraft is the amount of like ancillary options or, or I would say, services that they offer where there’s still room for more growth.

Like, like Minecraft is really, really popular, especially with, like, younger, younger demographics.

But there’s still so much audience across the entire gaming world that’s still yet to be captured.

You know, there’s this concept of more multiplayer type of play.

There’s this concept of the marketplace, which we have where, if you can imagine it like, like we’re, you know, for example, we just put out a Superman DLC, like, you can take any IP, like, any, anything your imagination can come up with, and you can create it as a game within Minecraft itself.

And that’s part of the magic of it, right?

Like, it really helps creators and even players who just love playing what creators are doing.

It just creates this, this infinite world of possibilities.

And so that’s why I really love of being here, is not only the past success, but the the possibilities and the opportunities that that we still haven’t even, you know, haven’t even started to tap.

[0:11:58] Matt Wright: It’s funny.

It’s awesome.

I actually think I was chucking to myself too.

Like, the working for Minecraft is probably, like, one of the best, like, your dad’s a superhero, or something like that.

Or your parents a superhero.

Like, great, great plan for acts for your kids to be like, Oh yeah, that’s my my parent works there.

So that’s awesome.

Do you think you could maybe just share a little bit, especially in the realm of, like, experimentation, like, what’s your like?

What are you actually focused on, from like, a key business outcomes perspective, or in terms of like, what you’re trying to either design to happen or optimize, like, like, what does that look like at Minecraft?

Section 5 — Business Outcomes & Experiment Focus (Web, Marketplace, Gameplay)

[0:12:30 – 0:16:19]

[0:12:30] Tim Mehta: It really depends on the team.

And so we have our more commerce focused teams, like the website, like the marketplace, where, you know, in the end, you’re you’re trying to drive revenue, you know, and it, it does start with kind of what’s existing today.

And how can we optimize the journeys?

How can we optimize discovery to, you know, to drive more sales, drive more revenue.

But then there’s this other part, like I mentioned before, where there’s just so much more opportunity to, you know, to find new ways to help users not only discover but engage with the with the game itself.

And so I, you know, we talked about this a little bit before, but I also work with the teams in Stockholm, Sweden, who are much more focused on the actual gameplay.

So think onboarding, you know, you the UI team.

These are also teams where their focus isn’t revenue, right?

Their focus is retention or gameplay hours.

How can we help users, especially new users, find that, that fun, that value, that you know, that you could call it that Aha, moment sooner, so that they’ll come back and want to keep enjoying it.

When, when you think about the the the concept of what Minecraft represents as a sandbox game, there’s a lot of people who who don’t like being dropped into a sandbox and and having to figure everything out for themselves, like there’s a lot of people who liked that, who liked that more guided approach.

Of, hey, give me a tutorial, give me levels, give me missions, right?

Give me Give me more guidance.

But then there’s the base the game, like the the base players of Minecraft, who really love it, who they appreciate the ability to to truly start to just discover things on their own.

And so for outcomes of what we’re looking for, it’s, it’s very much a mix for me.

On the experimentation side, our mission is very much like to keep it simple, right?

It’s not, you know, hey, our mission is to drive revenue.

Our mission is to drive retention.

In the end, our mission is to use data in the form of experiments, in the form of user research, to help product teams make better decisions that are more impactful.

That’s it like whenever I I’m talking to people at at Minecraft and kind of.

Introducing them to this idea of experimentation.

That’s really what I what I try to land on is the goal is kind of like your R D department, like I use this with this example, or this metaphor, where you think about Batman, right?

And you think about he has this R D department who’s running experiments, trying to figure out what tools or devices can help Batman be more efficient, make better decisions.

And so I use this, this meme as an example all the time of Christian Bale is Batman and Morgan Freeman running his R D department.

And it really helps product teams truly understand okay, so experimentation is not this.

Hey, let’s go run some copy tests.

Let’s go de risk this feature.

But it’s very much this mechanism for product as a research and development mechanism to help us understand what are the best approaches we can take, right, like, what’s the best way we can drive impact?

And so I see that as much more the outcomes that an experimentation program should be focused on is, how do we help product teams make better, more impactful decisions for whatever their OKRs or important outcomes are?

[0:16:19] Matt Wright: If you then break it down on, like, okay, it sounds like you’ve got the website, you’ve got marketplace, you’ve got in product, right?

Like, those are all kind of things, like gameplay, in terms of like, you want to have, like, what’s a great onboarding experience?

What’s a good retention experience?

Can you maybe just give us, like, high level like, what does an experiment look like in each of these places?

And maybe with a little bit of a grain of like, what are we looking to, like, refine there, and maybe what’s like, tricky about it?

Like, I always think that’s an interesting question.

Of like, Hey, we’re trying to optimize for this, but there’s always like, a little bit of a catch, or there’s always like that, that difficult thing, and like, the trade off, or something like, would that be possible even just make that super granular,

Section 6 — What Experiments Look Like (Web vs In-Game) & Spillover Risk

[0:16:19 – 0:23:29]

[0:16:57] Tim Mehta: absolutely so with these more infrequent experiences, like the website, like the marketplace, even to a certain extent or onboarding, you have a lot more affordance to run more noticeable experiments.

So think manipulating the UI in a way that is very noticeable for users or even for the website, trying to sell totally different products that are totally marketed, totally different ways within the game, and you start to get to any product, you start to run experiments on people who are frequently visiting,

The same interfaces, the same pages.

When you take this step that direction, you have what they call a spillover effect with experimentation, a spillover effect, essentially, at least the way that we define it, is when two different users get a different treatment or different experience, and they notice it, and they start talking about it, the Minecraft community, very vocal community on social media, forums, on the internet in general.

And so the risk we run with running experiments within the gameplay is that people will start to speak up and say, Hey, I’m getting this different experience than my friend.

Is what the heck’s going on.

This is confusing.

This is frustrating.

Or, hey, we had this experience and now it’s gone.

Say we ran a test and turned it off, what the heck’s going on?

And so there’s different techniques and approaches you take once you get deeper into any product that is more frequently visited, and you need to make sure that you’re mitigating the risk of this, this spillover effect.

It doesn’t mean that you don’t run tests.

That’s the most important thing to understand, and a lot of products are skeptical and scared of running tests because of this reason that users are going to notice, and then they’re going to miss the treatment, or they’re just going to be confused or frustrated.

You’d be surprised how few, how many times that a test will actually happen in one of these kind of environments, and people either don’t notice or they don’t mind or they don’t care.

Like, I know, I’ve worked with a couple different teams at Minecraft, where I asked them, you know, when they brought this risk up, like, Hey, should we run this test?

What if users notice?

And then, you know, they get confused or they complain.

And then I asked, I just said, Has this happened before?

And then the team kind of just looked at each other and said, Actually, now that you mentioned it, no, we’ve, we’ve never had this problem where people actually raised enough concern that we had to do something about it.

And so we do have, you know, obviously human as human beings, especially within your product, we’re a little more sensitive to risk in the sense of where we may be, a little less motivated to run experiments.

Could cause this spillover effect.

But in the end, if we’re really focused on improving the experience, not just for driving business metrics, but for the user themselves, experimentation is just really, really valid, like vital, it is critical to continue to do that.

So we do have a different approach with the, you know, gameplay teams, and these teams who may experience this little bit more, where we teach them different ways to reduce this risk.

So you just run tests to different cohorts.

You reduce the actual sample size.

Like I said, Minecraft has a lot of people, so even if you reduced it down to 20% of your total audience, you’re still going to get significance within a reasonable timeframe.

We also have a patch notes that any game update really has, or any product update really has, including an experiment you’re running in the patch notes is also another idea or a way to reduce that risk.

So if a user does realize they’re in a test or that something’s different, they’ll go to the patch notes and see, okay, they’re running experiments for X, Y and Z.

And so this is, this is probably one of the biggest, I would say, nuances of, you know, moving from these more single, single frequency experiences like onboarding, like buying the game itself into these very frequently visited experiences.

And you do have to be much more careful in the approach when you get deeper in the product.

And so your velocity is not necessarily going to be the same as these marketing you know, these acquisition type of funnels you may run, you know, a lot less tests you know, or the magnitude of the change for an AB test may not be as big as you’d like, but in that case, you really just lean on those things other than an experiment, that are going to validate your decision make, help you make better decisions.

So user research, that’s a that’s a key for any experimentation program.

I consider it really the fuel of of any any proper product development team.

You know, what are the pain points our users are actually having?

And then how can we flex our own business intuition and creative problem solving skills to build a solution for these validated problems.

And so leaning a little bit more on that when you get deeper in the product is also helpful, because then you’re not investing and taking this risk in something that’s maybe just a gut feeling.

[0:22:34] Matt Wright: I also, I think, like, the thing that comes to mind when you describe the example of, like, going to your colleagues and asking, Hey, like, have we seen this happen in the past?

I think that’s like, such an impressive like, or big muscle we can pull on, like, asking the question of, have we seen this behavior?

Is there evidence that people are, like, worried about this or, like, that risk is real, and just as much as it’s like, we need evidence to have good tests that we want to run, or that’s like, a better practice that we should go about, I think the reverse is also really true of just asking the question about, have we seen actually, this thing we’re worried about in practice?

And if we have, that’s actually great.

But if we haven’t, we should be almost testing for that as well.

Like, even if it was to have complaint or something like that, well, at least no, that’s like, people are sensitive to that.

There’s elasticity.

And that in itself, I think, can push us in whole bunch of like, interesting directions.

It’s like, super powerful.

[0:23:22] Tim Mehta: It’s so funny you said that because that’s exactly what we suggested to that team.

Like, okay, well, let’s run a couple of tests and see if it happens.

[0:23:29] Matt Wright: Love it.

I’ve heard you talk about this idea inside of Minecraft called where you’re focused on driving measurable growth.

And I think that measurable growth is kind of like an interesting one.

I’d love to just maybe just double tap on that and ask, like, what does that actually mean in practice?

Like, what are you meaning when you say that?

Section 7 — Driving Measurable Growth & The Ladder of Evidence

[0:23:44 – 0:30:12]

[0:23:44] Tim Mehta: So there’s two approaches to a product organization.

There’s a delivery approach, where every product team, they’re given a feature, and their goal for the next quarter, two quarters, is to deliver the feature.

There’s an approach of a more product type of mindset, and I would say even a growth mindset, where your goal is not necessarily, here’s a feature, go deliver it, but your goal is, here’s the metric that your team owns.

Go do whatever you and your team believe is most valuable to improve it.

And yes, features are a main mechanism in order to drive this retention or revenue, or whatever it may be.

And so when we talk, when I mentioned, drive measurable growth, it essentially means going from this more delivery, fixed mindset of here’s a thing, deliver it, go on to the next thing, over and over and over to here is an important part of the product we need to improve, or important part of the experience.

Let’s in a measurable way, understand what opportunities will actually drive the metric, and then go build and deliver those and doing it in an incremental way where, just like I mentioned.

And before, you’re leveraging experimentation in every step of that product development cycle.

Otherwise, if you’re not driving measurable growth, you’re essentially taking very big risks and incurring sunk costs on a very frequent basis.

I mean, I’ve been at a lot of different places, and I can’t tell you how many times I’ve seen an investment into an idea that turns into a feature and a focus that was very much from gut feelings, that was like, hey, you know, leadership believes this is what we should do.

Let’s go do it.

And when those things are delivered, and, you know, we don’t see the impact we’re hoping for.

That’s a mess.

That’s a mess for everybody versus, you know, even at the very onset of an idea on the business strategy side, we can go and test things to validate, okay, of these business strategies we want to go after which one actually shows the most promise.

And you can use experimentation.

You can use your user research to actually validate those things and then go after them.

And so driving measurable growth doesn’t mean, hey, incremental little tweaks here and there, over and over and over, that’s just part of it.

That’s maybe half of it.

The other half is very much.

Let’s validate these bigger bets before we invest in them.

I actually have a visual that I use with my teams, and it talks about the likelihood of impact based on your supporting evidence.

Gut feeling is, I have to do this down here.

Gut feeling is usually the lowest right, like, if you’re just, hey, I don’t have evidence or data to prove this, I just feel like this is the right thing to do.

If you invest from that point, your likelihood of impact is at the lowest it could be.

But if you start to go up the ladder of evidence and start to collect more data on the idea, then your likelihood of impact starts to increase, right?

So if I start with a gut feeling for you know, maybe it’s a new feature or business idea.

And instead of going and telling the teams to invest right away, I ask some of the teams, hey, let’s go do Can you do some market research?

Can you do some user research?

Can you run some tests and tell us, is this an opportunity worth going after?

That’s when you’re kind of climbing the ladder of evidence, and you’re maximizing your likelihood of impact as you get higher.

I also have, I know I’ve shared this on LinkedIn a bunch, but everywhere I’ve been right, like, I’ve been at Microsoft, you know, worked at Lucid, you know, root insurance and then on the agency side, work many different places.

Whenever we have a test idea that comes from user research, so it’s targeting a valid user problem, we tend to see a 50% success rate.

And I’ve just kind of been surprised that that number, that 50% number, has persisted across all these different places I’ve worked, whereas the industry average, which is which is pretty well shared across the industry, is usually between maybe 15 and 30% and that doesn’t matter how high of maturity your program is.

That’s just it.

[0:28:19] Matt Wright: Yeah, totally.

It’s funny.

You said two things, I think, on the hierarchy of evidence, note where that goes up.

I love also the fact the idea of like, you have better likelihood.

But I think the other element on that side of the graph, I always liked was the degree of bias, which you move away from just like, an opinion based of like, what does someone think to we do have some supporting evidence, or it’s fact based, or those.

And just to add to your point of your anecdotal we literally have literally built repositories of experiments.

And sim, who did a lot of this work, actually did this really cool meta analysis where he took what is backed by evidence versus things that were just like experts in the field.

And it’s like pretty amazing.

It’s almost exactly you describe, of like one in three tests winning to jumping to like, a 50% win rate.

Very much.

Like holds true.

And I think that’s something I heavily, heavily just encourage everyone is to really like, explore, how do we look at like?

Is this an evidence backed idea?

But then I think a very simple thing too, to maybe build on what you’re saying, is like, look at the concept of iteration of like as you run experiments, and kind of like, what we’re saying we’re reflecting on like, what is the evidence we learn from them?

How do we supplement them with additional research or like findings?

I think that’s some of the really, really powerful stuff that we can get out of it, or off the back.

I think it’s like, amazing how easy that is.

Is like a muscle to pull.

Okay, second one here that I’d love to tap into, because I think this is really cool.

Is really cool.

I think you’ve mentioned the idea of the power and the importance of your experimentation platform to enable experimentation, to drive impact and build a culture really around like the process of test and learn doing experimentation.

I think you have three key principles for these that are really cool.

And I think I love it coming from you too, that you’re very like.

In a space of not tied to one tool.

You worked across a whole bunch.

Could you share what those are?

And then maybe just like, given a like, why are those important for the audience?

I think these are great as someone to think about their platform

Section 8 — Three Principles: Trustworthiness, Velocity, Democratization

[0:30:12 – 0:37:13]

[0:30:12] Tim Mehta: for sure.

So any organization you go to, it’s your actual tactical approach of experimentation is going to be different the problems you need to solve, the areas you need to focus on.

It’s never, there’s never a template, like a plug and play template that’s just going to work your operating system, which I know this is how a lot of different places use that phrase.

It needs to be very bespoke to your problems and your opportunities.

The thing that persists, and I think this is where I’ve seen, you know, most of the success I’ve had stem from are the principles.

And so these three principles, it works for, just in general, your experimentation program.

These principles apply to your platform that you’re using to drive experimentation, and anything, even in the product development realm, these three principles can be so key.

So the first is trustworthiness, the second is velocity, and the third is democratization, trustworthiness.

It essentially means data integrity.

Are we making decisions based off of valid and trustworthy data is the data that we’re looking at trustworthy, right?

Like this is usually the very first place you want to focus are the metrics that we’re tracking actually coming through with high integrity.

Do we have high trust that we understand what these metrics actually mean, do we have high trust that these metrics are actually meaningful and valuable to the business, to the user?

It also means, kind of what we were just talking about, that we aren’t leaning too much on unvalidated data approaches or opinions.

So trustworthiness just as much on the quantitative side lives on the qualitative side.

So if someone pitches an idea, especially leadership, they’re very smart about this, they’ll ask, Why are why is this idea worth going after?

And a lot of times with these CRO programs, it’s really based in a gut feeling, right, or we saw a competitor do it.

So that trustworthiness piece very much also translates to is this a validated user problem?

Can we trust that we are investing in this area based off something real?

And so that’s what trustworthiness really means.

And then velocity, velocity, it’s very straightforward, and this is how quickly can we learn and deliver I have recently listened to a podcast with the CEO of Duolingo, and he said, you know, he was asked, What strategy would you recommend to any new startup?

And the very first thing he said is, well, you need to run a lot of experiments.

And Duolingo is an incredibly successful product that is based and baked in through that principle around running more tests, 80% of our tests are going to fail.

You know, we talked about this rate just a few minutes ago.

So if you ask any statistician, how can we drive more impact?

Knowing that particular data point, they’re all going to tell you the same thing.

Run more tests.

Now, remember, trustworthiness is your first principle here.

And so if you’re running a lot of tests that are low trust, nothing’s going to matter, right?

You’re just kind of going through the motions of it.

And so it is important to remember all of these principles are, are really almost in a sort of order of importance, trustworthiness, then velocity and then democratization.

And velocity is not just engineering either, right?

It’s like we have so many processes, from the handoff of design to engineering, from the ideation to actual, you know, test or tickets in your in your platform, there’s so many opportunities to improve velocity.

AI is a great example of how you can do this, too.

I’m sure we’ll talk about this in a minute.

But there’s, there’s usually almost infinite possibilities when it comes to improving your velocity.

And then democratization, essentially.

What this means is, can we take the other half of the value we get from every experiment, which is the learning, and scale it through the rest of the organization in the future, and now?

So think about having a repository like you mentioned, of all of our successful or experiments that weren’t successful, how can we make sure that every test isn’t just, hey, here’s the results, let’s go do it and that’s it.

But instead is, here’s the results, here’s what we learned about our users.

Let’s.

Share this across, you know, across the relevant teams who want to see it, or would get value from seeing it, and let’s document it so that teams, you know, for now and in the future, can go back and understand what we’ve learned and not make the same mistakes or capitalize on the same success that we’ve seen.

The other part of democratization on top of just documenting things well, socializing things well, is enabling teams, right?

Not every organization can afford to have some type of centralized experimentation program who’s going to continuously, you know, take advantage of the tools for the product teams.

Democratization is enabling and giving others, not only visibility but autonomy, to leverage experimentation, user research and reach this level of learning and success themselves.

And so velocity and democratization actually overlap a little bit here, because you want to build tools that people can use really quickly and easily with low friction.

And you want to enable product teams to be able to do this on their own without needing some type of assistance or help.

And so this translates, like I said, not only from the program level, but the platform level.

So you know, you think about any experimentation platform if you can tick all three of these boxes, trustworthiness, I trust the data that this experimentation platform is pulling in and the output of the results from this platform, velocity.

I trust that this tool can actually deliver experiments or feature flags in a high, high high velocity way without hurting the integrity of my my infrastructure and democratization.

Can many people comprehend and use this tool easily?

If you are missing any of those principles with a platform, adoption is going to plummet, right?

It really takes all three of those things, from a program and platform perspective, to truly find help teams find the most success they can from experimentation.

[0:37:13] Matt Wright: I understand, agree, I do want to just double tap into this a little bit too.

Where have you seen like, people like, I guess across all three, like, maybe give like, where it’s like, hey, you’ve seen the blunders.

There’s mistakes that, like, what does that look like when it’s not going right for each of them?

Section 9 — Common Mistakes With Trustworthiness & Tooling

[0:37:13 – 0:42:39]

[0:37:28] Tim Mehta: So for trustworthiness, I think one of the most common ones I’ve seen is ideas that come from baseless, baseless thoughts, or or baseless, you know, concepts.

So that goes back to that supporting evidence piece.

Are we making decisions based off of high, confident data?

And so think about, you know, when you’re coming up with test ideas, say you’re in a brainstorm, a lot of those are gut feeling ideas, like, here’s our flow, here’s what we own.

What do you all think we should do?

And then people come given their ideas, and 80% of them are just gut feelings.

I see that all the time.

And the other part is the experimentation tool.

I mean, you know, there’s lots of tools out there where I see a lot of people just kind of ignoring the output, like a lot of tools will tell you, you have this much sample size.

Here’s over the direction your metrics go.

Here’s the level of significance.

But there’s so many organizations that just they don’t trust it.

They say, let’s just look at our data warehouse.

Let’s look at our own data and do this whole calculations ourselves, because we, you know, we have higher trust there.

So those are two examples for trustworthiness where I’ve seen, you know, probably in the most common terms.

[0:38:50] Tim Mehta: You know, the challenges there.

[0:38:52] Matt Wright: Do you think there’s a way to like, like, fix that, like, just curious, like, doubling in on the don’t want to go too deep on tech and tools, but I think maybe I’ll just ask you, like, does it matter?

Like, should it be better that you’re doing the analysis, like, deeper in the areas you trust, or is there a trade off?

Like, the first thing that comes to mind is, if you don’t trust what’s in the tool, it’s like, yeah, you can do the additional analysis, but that’s essentially, actually not making it available to everyone.

You have to go to another team and, like, ask, can you run this potentially, or it just can’t, like, have that, like, more, like, rapid, quick fire, like, like, like thinking.

Or is it just that’s the way it is in most organizations?

Or that’s just a challenge of modern measurement,

[0:39:30] Tim Mehta: I think it depends.

Like, if you’re using a home grown experimentation, you know, feature management tool, you have a higher likelihood of having that trustworthy outcome for more third party tools, where it’s not homegrown, I think it just takes investment, like I remember, I’ve spoken with some, you know, platform teams, and I asked them about this challenge where, you know, many, many, many organizations, they just do their own analysis.

They don’t really trust the outcome of the the tool itself.

And they.

You know, they mentioned to me, you know, we could do that, but it would, it would cost a lot, right?

It would cost, you know, a lot of product team time for them to make that happen for each customer.

And so I do think when platforms are able to say, or, you know, these third party platforms are able to finally say, you know, what we want to be the one stop shop feature management and analysis tool for an organization, and they’re willing to make that investment.

That’s when you start to see differentiation between tools 100% and so from a from a solution perspective, that’s that’s really it right?

Like, I don’t want to sit here and say everyone should have a homegrown tool, not, not every business can afford to do that, but I do think that is one big opportunity for these third party tools and vendors to to build that trustworthiness in the outcomes that these, these tests that they’re hosting are actually, are actually delivering.

And then the other side we talked about, you know, the qualitative side of trustworthiness, how to fix that.

It’s really making sure, instead of waiting for the pm to get in a meeting with a leader and the leader asking, why are we doing this?

Like, what data drove this decision that that becomes a muscle memory for the team itself, where, hey, we you know, instead of having a brainstorm of which you’re just like, here’s what we own, what is everyone’s ideas, your brainstorm is, here’s what we own.

Here’s some results from research we’ve done talking to users based off of all of these problems that users expressed, the most common problems to solve.

Now let’s start to brainstorm, right?

So you’re fitting in this second step of this ideation phase of not just here’s what we own, what should we do, but here’s some validated problems or opportunities that we have.

Now, what should we do?

And in the end, that’s that’s what any product team gets paid for, right?

Is like, let’s take problems and use our design creativity, our engineering creativity, our product management, business intuition, and create a solution for these users that are, you know, that’s coming from something that’s real, versus what we just assume.

Like, I think assumptions or business intuition is much more valuable when we’re flexing it towards a valid problem to solve, versus just a known place that we have ownership.

[0:42:39] Matt Wright: Now I love, I love that you’ve also got, like, a ranking or a prioritization of these, these principles too, because I can’t agree more about the trust being like you just need that fundamentally to be there.

That’s kind of like the bedrock that if it’s not, then everything kind of can’t flow from from that point on.

You you.