0:00 — Intro + Kevin’s role at Best Buy

Matt Wright 0:00
Hi everyone.

Welcome back to the conversion podcast.

Very excited once again today to be doing a conversation with Kevin Washington, former head
of experimentation at Best Buy.

Make sure I get that title right.

Kevin was that, what’s your actual kind of role at Best Buy?

Kevin Washington 0:15
What was the Yeah, so I was the director of product management over experimentation, and
just basically the head of experimentation on the data science function, the analytics
function and product management for Best Buy.

Matt Wright 0:26
Nice, awesome.

And then just kind of setting the scene for this one, Kevin’s been really awesome to kind of
maybe walk us through some of the mindsets and thoughts about, hey, how does someone go
about like an organization like Best Buy, thinking about how to run experiments on the site,
and what are they actually trying to test and make decisions about?

We’re going to try to set this up a little bit, actually, as a bit of, like a screen share
discussion.

So ideally, if you’re watching this, if you can see it in your podcast player, like Spotify,
or we’ll make sure there’s, like, a YouTube feed.

This would definitely be a visual one that would be awesome to check out.

But with that, maybe Kevin, if you just want to, like, kind of tee this up a little bit,
maybe you just, like, kind of share your screen and just kind of walk us through, okay,
foundationally, how does like a company that Best Buy, think about, like, optimizing its
type of web property

1:14 — How Best Buy makes money (conversion, ads, marketplace)

Kevin Washington 1:14
100% I mean, when you think about best buy in general, you think about, how are we making
money as Best Buy?

I’m trying to push push purchase conversion, and I’m trying to push ads.

Those are kind of my two primary revenue streams.

When you think about competing, especially right now, margins, as thin as there are and
everything else, especially for Best Buy, you’ll see that they’re pushing more marketplace
type technologies, which is a different type of revenue you don’t have the inventory, you’re
kind of more competing directly with Amazon and then that retail media network.

So if I had to say they were making two bets just from what’s publicly available, I would say
Lisa Valentino and the retail media network space there and then on the marketplace side.

And how to recompete and get more Steve’s active and online.

Matt Wright 2:02
If I was to simplify it, would it be fair to say you kind of got like, like, three functions
of like this, like website running where it’s like, actually Best Buy, and it’s like, their
online retail environment, and it’s like, hey, I want to buy products from rest Best Buy, or
get me the right store to buy, like, a product.

But then, like, you’re saying, you’ve actually got a marketplace of other sellers, and then
at the same time, then you’ve got, like, the ability run ads inside of Best Buy and actually,
like, monetize some of like the ads.

And all those are probably a little bit intention with each other in terms of, like, hey,
how do we design the best experience for all three of those at the same same time, 100%

Kevin Washington 2:37
and then you got fun things like, you know, sponsored product placements, which are very
similar to ADS, right?

They kind of run through the same retail media network space, and then basis points on credit
card transactions through the Best Buy card and sign ups for that and whatnot.

So it’s there’s a good amount of different things you kind of have to compete from a metrics
perspective,

2:58 — Website priorities + “placement is what you own”

Matt Wright 2:58
Yeah, crazy.

Okay.

And then just so on the website itself.

Is there particular, like, we call them areas, but I’ll be curious Actually, your verbage of
how you think about it too, certain areas where, like, the retail one is trumping marketplace,
for instance, or the ad network or something like that, is, like, how do you guys actually go
about like, thinking about that and like, do you think about that in terms of, like, trying
to create areas and zones and priorities?

And I like the priority stack analogy, like, in particular, less important thing that we’re
trying to

Kevin Washington 3:30
trying to do here 100% I think the it depends on who you asked.

The idea, especially with Best Buy, is that there was this kind of good tension around retail
media is trying to drive this, but let’s make sure it doesn’t hurt that.

Okay, hey, the merchant leaders are trying to push x.

Let’s make sure it doesn’t hurt y.

Because of the margin conversations, I think marketplace and retail media network are really
winning a lot more of those conversations these days, but you’re not going to see a lot of
marketplace content on like, let’s, for instance, our homepage right now, for there, there.

I’m not there anymore.

But the reason for that is that the marketplace is just starting up, so they’re accepting
vendors and doing all that kind of stuff that’s all very public at this point, but you’re not
going to quite see that yet, but the way that we thought about it as we started talking was,
okay, cool.

If we see an increase in folks that are driving we’re gonna go drive traffic to our
marketplace.

What does that do to our existing relationships, existing vendor partners, and the dollars
that we’re making there?

Things like inventory turns still matter right at the end of the day, especially when you’re
thinking about Best Buy, like you got the 4p of retail right Price, Place promotion, product
is the important one.

Right as the E commerce like engineering team as the product team that owns the site.

The only thing that you really own end to end is place you.

It’s the placement of the product.

It’s are you showing it to the customer at the right time?

So that’s a lot of what the optimization within this is for.

And so that was another big conversation that we had a lot where, okay, cool, we’re going to
optimize for purchase conversion, but you can’t control everything.

And so everything’s very category specific.

Everything’s very user specific.

Randomization helps with a lot of that, but you really have to understand that just because
there was a change and it looks causal, there’s so many other factors that you have no control
over whatsoever that you really have to make sure you understand so de biasing was a big piece
only,

5:40 — “Job of the homepage” + ad placements + personalization

Matt Wright 5:40
yeah, no, makes ton of sense.

And then, so, if we’re on the homepage right now, so these are just like, Best Buy retail
products, then what mainly is it like, kind of showcasing, or kind of like, what’s, what’s
the job of the homepage, at least in how you think about it?

Kevin Washington 5:57
So, yeah, so I love that you said that job of the homepage.

Job of the different pages.

Something huge about the idea here is to drive you deeper into Best Buy what you’re going to
see as you’re on the page are ads that are going to take you, even if they’re ads, all of
these are sponsored ads.

They’re going to take you to another Best Buy page internally.

So we’ve got Oakley meta glasses, which I did not know existed until right this second, paid
for that ad to be there.

They’re paying about clips per million, but it’s still best buy.

So when you purchase here now, I’m getting paid on that side of it as well, because you’re
still within my ecosystem.

Matt Wright 6:39
That would be, like an ad network placement on the homepage in that case, right?

That would be drive Best Buy.

BLP, in this case, right?

That’s like, sort of the where that click would go, yeah, exactly.

Kevin Washington 6:50
And so if you look directly, there’s five different ad placements on the site currently.

So you got this banner across the top.

You’ve got these.

Apparently, this is not an ad fees here as well.

And at the end of the day, a lot of this homepage specifically is about personalization.

We did a massive effort and we wrote some blog posts and whatnot about it.

So I think we could talk about it without an issue around actually, sorry.

Let me share to the screen, my apologies.

So you can see here we have five different ad placements across the site.

Top here.

These are here.

And so then once you’re even in here, like these are, again, more ads.

And it gets really difficult to tell whether something is an ad or a actual, just Best Buy
link that’s really by design, because if we come in here, we search laptops,

7:53 — The hard-to-test problem: ads pulling clicks away

Unknown Speaker 7:53
a good example of A thing that was difficult to test

Kevin Washington 8:01
this ad right here, so we’re recommending additional products, but you’re already searching
for products down here.

So by adding this banner, I’m driving clicks away from something else.

And so I want everybody to click on this, because that makes the retail Media Network dollars
and Best Buy dollars, but now this team is going to start to see their clicks decline.

Are we okay with that?

How okay with that?

Are we and how do those metrics kind of drive into each other?

Is a constant conversation.

Matt Wright 8:37
If we just got a little granular on that one specifically would be the primary metric on that
type of like, like test, yeah.

8:46 — Primary metrics vs guardrails (revenue vs add-to-cart vs conversion)

Kevin Washington 8:46
So if, as we’re adding it, it would be dollars that we made off of this ad placement versus
reduction in like add to cart rate, for example.

So the guard rep, so we set up a primary metric of number of clicks on this and how much
money we actually made, as the one thing that was always an interesting tension, because the
retail media network team can show real dollars from a quick test,

Matt Wright 9:16
yeah, as soon as someone clicks, you get Your direct revenue attribution, right?

Yeah, yeah, yeah.

Kevin Washington 9:22
And a really fun thing, actually, that we did with AB tests was we would run a test for,
like, let’s say a new client.

Bose is obviously a client, but let’s say that they’re considering go into some new pages or
something.

We can go run a test and show you how much revenue you could make, and then reference it
with, you know, full AB metrics and P value and all that kind of stuff.

So that was something I was really bullish on as I was moving out.

Was, how could we give like, the vendors more control over their own testing and whatnot?

Yeah, but if I were going to set this test up, I would set the primary metric up as, you
know, Rev.

New from puts per million, yeah, and then I would have that secondary metric of not harming
Add to Cart rate by a certain amount, because there has to be a balance there.

And trying to do some, you know, profit sensitivity analysis on Okay?

Well, if this goes down by this much, how okay are we?

And then purchase conversion really being that guardrail metric, because even if you’re
clicking on this, if you’re not converting, there’s an assumption in that case, that we’re
showing you something that you didn’t actually need, and we’re actually driving you away from
the thing that you came here to do, which makes Best Buy less desirable company in the long
term, because we didn’t give you what you need.

And so you might have clicked on this ad suite, but then you went to Amazon because you’re
like, Ah, this doesn’t have it.

I’ll just go on chat GPT and ask it what the best computer is, and just follow whatever link

Matt Wright 10:53
I get.

I ask, like, just, I mean, super fascinating, too, that, like, you’re choosing the these.

I’m just gonna call these, like, the retail products that Best Buy owns, versus ad network
dollars.

Like, like, how do you walk through that?

Is it just a volume game?

Is it too far down the funnel to make a relative comparison?

What’s a good way to, like, think

Kevin Washington 11:11
about that?

That’s really the way I think about it.

I think that there are people that would argue with me, but there’s just so many, there’s so
many teams at Best Buy, and there’s so many people that are influencing different things at
different times.

And I’m making a change.

Exactly what you said there.

I’m making a change on this page.

What is, what can happen on this page that is no longer going to happen after I make that
change, I like to say as close to the change as humanly possible to get a real apples to
apples comparison, because if we say, Okay, well, I’m going to do just Well, that’s why I keep
purchase conversion as a guardrail metric.

But you can do purchase conversion on the retail Media Network clicks for this ad versus
this, like whatever shows up in the search recommendations, but you’re then so far down both
of those paths, that’s kind of that, is it actually you’re going to get the correlation,
like, how quickly?

And like, sorry, get to more than a correlation and how quickly.

So it’s, for me, it’s the thing that’s happening here is you’re clicking on this ad, and then
the thing that’s happening here is you’re clicking on this Add to Cart.

Yeah, and just

Matt Wright 12:22
makes Bob’s your uncle.

It’s what you what the actual action is directly in the relative test comparison.

And then, like, you take secondary effects, and then kind of look at, like, how, how that
kind of plays out for them.

Like, super fascinating.

One of my favorite

12:37 — A/B testing pitfalls: “failure” can hide funnel wins

Kevin Washington 12:37
examples is, when I first got here, they’d run a test where they had people landing on kind of
the search page.

It was kind of an SEO optimization test or what have you.

Yeah.

So people were coming here, and they had made some changes to the way this lit.

This was laid out.

I think they went with this versus kind of the bar list graph.

So what they did was they did was they only looked at purchase conversion, because that’s kind
of how Best Buy operates.

There was a, I want to say, like a 40 basis point increase in add to cart rate, and there was
a major uptick in number of customers on the that were hitting the page.

So not only did you boost traffic, you also boost engagement, like wholesale, yeah, but add to
cart rate went down.

And add to cart rate went down because of their because there were so many more customers,
that the rate went down, but the absolute didn’t.

But there was an issue in the fact there was a thing that was confusing about the checkout
page.

Remember exactly what it was in both experiences, but you had so many new customers that never
shop on Best Buy that were in the test crew.

And I’m hoping making sense is on that once they came to the checkout, they were even clicking
checkout and getting confused and leaving, and that was where the bounce was happening.

Because even right here, it’s like, okay, well, now you added this extra step.

I know that you’re a new customer, or should generally a lot of the time.

So like, do we do we need to have this and like, how can we give an expedited checkout if you
come through this funnel, but they call the test a failure.

And this was just, I just happened to be going through all the details of the different tests
that had happened over the last year when I first got hired.

And I said, Well, what happened with this?

There was no connection to any of those good metrics that they boosted, because the success
or failure was based solely on whether Optimizely told them that they won or they lost based
on purchase conversion and yes, at the end of this what we’re trying to drive towards, but if
I just hacked so many additional people into the funnel, and I’m getting them to the next
step, and even the next next step in a lot of those cases.

And then they’re falling off.

There might just be that’s a new cohort that I haven’t seen here before.

And now I need to do something to try and, you know, attract them like it’s an Eric Reese
thing from the good old days of everybody had to read his book, right?

Lean Startup, exactly.

Metrics are people too.

I remember reading that is a we product led, but it’s really taking and understanding the
customer journey.

It’s not just about the raw data and what you what you think, you know, there’s still garbage
in, garbage out, when it comes to tests up and everything is

Matt Wright 15:32
and just, just to clarify, too, for people listening, what?

What in this case, was driving, like, a new cohort, like, was it the at the ad itself was
just getting in a whole new group of people actually, like into that PLP or PDP?

In this case,

Kevin Washington 15:48
yeah, we had done a AB test on a type of marketing campaign that was like sem driven, and so
it was boosting people to this page specifically, yeah?

Because otherwise we didn’t typically drop people in search, yeah, usually we drop them in
other places.

And so it was like a specific view of search.

It had really high add to carts and and whatnot.

But yeah, that was, that was probably one of the most interesting insights, and that’s one of
the reasons, and that’s one of the reasons that I said, Okay, we’re just not going to use
purchase conversion.

Oh, look, I hear you.

Yes, that’s a thing.

We’ll keep it as a guardrail metric.

But what we really need to start to do is look how are the things that we’re doing generally
affecting and here’s one more piece that was that’s important to understand why I didn’t do
purchase conversion, the changes that we were making were so small that the likelihood that
you were actually going to make a meaningful enough change to see a result was so so low
outside of the page that you’re to say that you’re going to be able to go multiple steps down
the funnel and still have an impact.

It wasn’t just unrealistic.

And so instead looking at something more like a Bayesian regression or something over time,
where, okay, cool, I’ve got these informed priors.

I know these people were in this cohort at this time, and I can really understand how the
changes that did go into production then started to drive additional value in a way or the
other, having that regression that looks across everything and says, Okay, here’s the group
of tests, here’s the time that they went in.

And let’s look at the dollars that we made of just trying to extrapolate off of an AB test to
because that’s another thing that was happening.

I said, Hey, the AB test made.

We purchase conversion by 20 basis points.

So in revenue, If we extrapolate that over the year, it’s gonna be $40 million it’s like,
okay, cool.

Let me get to the end of the year, and everybody’s looking around like, so what happened all
that money?

Because if if the ecom team made all the money they said they were going to, would be an $88
billion company instead of $44 billion

18:08 — Short-term revenue vs long-term trust, returns, regret

Matt Wright 18:08
so I was gonna ask you diving into this, actually.

So you’ve got kind of different revenue drivers, which is kind of fascinating.

You got like the media, the media network is going to drive revenue now, in addition to like
the team, put marketplace aside for seconds.

You’ve got different I think, I think you kind of were mentioning it.

How do you think about like, short term versus like, long term implications when you’re doing
this as well, right?

So, like, it’s like, we can drive short term, like, like, revenue.

Like, is there anything to like, consider there?

And I wonder if there’s like, I think that’s where it gets in, a little bit more of the art,
rather than just the pure science.

Like, it’s like, how do you think through that?

Kevin Washington 18:44
I think that’s exactly right.

CART size is a big thing, right?

Returns is a big thing too.

There’s some, don’t call them dark, but there’s UI patterns or some approaches to things that
you can do to get people to buy things now and then regret the decision later, being really
at that right place, at the right time, at the right price, at the right promotion, right to
provide the value that somebody is looking for in the moment, I think, is really what we’re
driving Ideally.

But how do you mechanically kind of determine that revenue and whether or not you’re giving
that to a customer.

We did a lot of things, like trying to determine who our best customers are based on a set of
actions, so how engaged you are with Best Buy over time.

And there were, like, kind of these behavioral values that we’d assign to different actions,
and then it was a question of, do those people take those actions because they’re their best
because they’re our best customer, or are they our best customer because they took those
actions?

So that was kind of a fun exercise.

Desires to do a series of AB tests to determine if we pushed people that were not our best
customers to go take those actions, would they then become them?

Is this one of our like ideal experiences?

As far as like this is defining for Best Buy, solving your job to be done, or is it not?

Matt Wright 20:11
Would you define that?

Like, what would be like the how would you measure?

Like they are now, they’ve gone from, let’s call it not an ideal customer, but have now become
like our best customer.

Like, how would you actually measure and then put them in that segment or cohort?

Kevin Washington 20:30
Yeah, it took time, but there was a weekly, daily, weekly, monthly, kind of stickiness, of
like, coming back, not moving.

The app was a huge thing.

If you had an experience that was X, Y or Z, and then download the app, and kept the app
downloaded for 30 days, and, you know, kept it open, turned on your notifications, engaged
with different things.

Were more willing to open an email.

Like there was a specific percentage at which someone would open an email from Best Buy, they
were a more engaged customer.

So there, there were some different things that we could do to kind of see over time, but it
was probably a three month endeavor of creating these different cohorts and try these
different things to push people into different actions to see which ones were actually driving
revenue at the end of the day, right?

Like LTV is what we’re trying to drive.

How do I get somebody to if there’s an assumption of X share of wallet, how am I capturing
that share of wallet?

And it could just be that they’re buying their small stuff here.

Hey, I need a new phone case, I need new cables, I need new whatever, right?

And then that lead, do those lead to a large purchase?

Because we’ve built the trust, because the end of the day, like, what do we have versus
Amazon?

It’s that we’re Best Buy is supposed to be where you go, because there’s expertise, and we’re
only going to sell you the top of the line, like best product.

And so it’s similar.

That’s something I did at Home Depot, where it was a question of, how do I recreate the store
experience that we’re known for online home depot is really project selling.

You have your project starter item, but there’s a reason that the dirt is in the back of the
store so that you have to walk past the flowers and the hose and the shovels before you get
to the things you came for, and so by then, you’ve already stacked up on everything else that
you needed, in addition to the dirt, because the project started first attached item basket
versus all that.

Matt Wright 22:18
Is there an analogy that you can just what does that look digital?

So you kind of talk about the back of the store.

Is that possible a digital experience?

22:32 — “Back of the store” digitally: recommendations + AI discovery

Kevin Washington 22:32
A lot of it’s like recommending an additional product that you might need.

That’s what you get a lot of the time.

And that’s where a lot of people kind of start and stop.

We attempted to resolve some of that with AI, where we had, we had the AI experience, where
you could sit there and chat.

And that really, I think, identified some additional interesting patterns, where somebody
would look for an item, cool, they’d find it, hey, what else goes?

What else would somebody that’s buying this like?

Right?

There’s a lot of gift finding.

So over Christmas last year, we did a gift finder AI experience, where you could give some
information about whoever you’re shopping for, and then they’d provide gift recommendations
and things that went along with it, and things like that.

And we found like the discoverability is really the place where that project selling or list
path, though, you know, solving whatever your need is, but giving you all of the things you
could possibly go along with it.

Another interesting thing along with that was the ads that you show, and the value of the ads
at that point where it’s like, if somebody’s buying speaker systems or the kind of vinyl
record player or something like that, then they might want, like, ear protection, because
they’re probably somebody that goes to a goes to concerts more often.

So there were some interesting things like that within the data.

I know we’re getting, well, experimentation, but like that, there was a lot of really fun
stuff.

Matt Wright 24:00
I think these are super linked.

I think they’re like, Yeah, I think the idea of segments and how they behave, and at the end
of the day, like, it’s like groups of people, and how do we, like, design the best possible
experience for that.

But in the context they’re in, it’s, like, very related.

I have, I have spent like, like, it’s kind of shit.

You talk about the value proposition of Best Buy, right?

I have used it before where I actually think, like, I have a lot of trust in it.

It has really good delivery.

I think their return policy, I had so much like, confidence, if it’s not the right product,
like, it’s easy, I can take it back to, like, a store close by to me is also feels great,
although I think with the sort of the network, or maybe I’m not saying it’s not, you know,
the marketplace, that I don’t know if that actually holds true anymore.

And almost like it makes me a little bit more wary of, like, Oh, can I take this back?

Like, what’s a marketplace?

Good, versus, like, like, a Best Buy, good, these different things, which can be a marketplace
you have, like, the media network, so I’ve got ad placements.

I feel like there’s a lot of really good stuff of, like, Why buy from Best Buy sprinkled
through the site that I’ve always met?

Like, when, like, thought about, okay, like, this is a really strong, like, experience,
building confidence.

Were all, were you able to maintain all that?

Were there trade offs where there was less prominence of that?

Like, maybe, I’m just curious, does Best Buy?

Like, think, is that even important?

Or it was just almost like, that was table stakes.

It was just

24:10 — Brand equity: “if we can make money, make money”

Kevin Washington 24:10
always part of it.

There.

It was an interesting conversation.

There were not really great metrics around it.

It’s the idea of brand equity is kind of what we got down to.

It’s we’re telling our brand story, and what’s the value of that in the marketplace.

But it was always, it was always secondary.

It’s like, if we had nothing else to put in that placement, then, okay, let’s do some brand
story stuff here.

Or there were specific places for it.

There used to get to be this Discover tab in general, where you could go and you could learn
about the different brands and why certain, why Best Buy picked certain brands and different
things like that.

That was,

Matt Wright 25:01
I’ve never seen that, so it’s like an interesting thing, but I’ve always been really skeptical
of like, I think the AI experiences, just at least not yet, have delivered, I think, as much
as, like, your intuitive, what we’re all used to in terms of in and out, like, shopping
behaviors, but sorry, I don’t want to derail so, like, just, just thinking about that, like,
in general, though, yeah,

Kevin Washington 25:22
I mean, and that’s, that’s really a lot of what it came down to was, if we could make money
with a placement, make money with the placement, we can’t make money with the placement.

Let’s tell our brand story, and let’s try and build additional equity.

Matt Wright 25:43
So almost that would be the priority, though.

It would shift to like, hey, there’s, like, this placements got more value than we’re
creating, and we’re actually gonna go seek to capture it.

Kevin Washington 25:57
And I mean, especially right now, the way that retail is it’s you’re not seeing as much brand
loyalty, especially from like, younger generations whatnot, and they’re buying a lot cheaper.

And so the reason for marketplace, if you think about it, phone cases, is a great example.

Best Buy has tons of phone cases, but there’s probably a million SKUs that they don’t have
access to, that people are going and buying, and they don’t have time to keep up, and then
they end up with all this extra on hand inventory because they’re not selling through whatever
color, because it’s not a thought right now.

So the idea behind the marketplace was, we just want more of categories that were, we don’t
need to really put our name behind to trust, like we’re not going to go marketplace a bunch of
laptops.

Was the plan.

Don’t know if that is the plan, but comes back, but it’s more of the long tail skews, which is
a similar approach to what Nordstrom took when they did their mark.

I was there for that as well, yeah, but they their whole thing, again, was we’re going to
curate the hell out of who our marketplace partners are, as if they were an actual vendor for
Nordstrom.

Because, again, that’s another company where it’s our brand image.

I realize I’m worked a lot of companies where their their brand image was kind of the, the
main selling point.

But, yeah, well, I was gonna ask you that.

26:50 — Marketplace strategy + SEO impact

Matt Wright 26:50
So you’ve got, then marketplace and curation and, like, maintaining the trust.

Is that the same, then on Best Buy, and is that, like, just very like, there’s subjective
person that’s, like, really trying to vet these marketplace sellers.

Or, like, how do they think through that?

And like, how to think of marketplace and like, maintaining trust.

Kevin Washington 27:19
My understanding, and again, this, this might be getting a little too behind the scenes, but
my understanding is that they were looking to be a more open from a like, number of sellers
that within a category, let’s say, but which categories they pick is the part that, I think is
a little bit more specific, right?

Like, they might not want to create a competitor in something like laptops that we’re gonna,
hey, we’re gonna open this up to any random whatever, and get a bunch of like, potentially
local level, quality things, because that’s, hey, that’s what we’re known for.

We don’t want to see a bunch of returns, whatever.

But again, with the idea of, like, phone cases, don’t really care, it’ll be fine.

It’s more about the fashion than the function, yeah, and maybe that’s kind of the delineation,
but, yeah, it’s a little sausage making, stuff that I wasn’t fully involved in.

I more just kind of helped with the Hey.

How do we think about as we add to marketplace?

How are we not hurting our existing business?

The idea with marketplace too, though, is from an SEO perspective, if you have more skews, you
have more chances to show up on Google, and so that means more eyes.

More eyes means more advertising views per million, clicks per million, which means more ad
revenue dollars.

So that’s how those things kind of start to tie together.

But it’s a question of as we’re getting those news, and what does it do return rate.

What does it do to LTV, weekly, active.

Of app utilization, downloads, things like that, because, again, you can make a ton of money by
one of the best examples was on the PDP page.

28:28 — Trust + category selection + seller openness

Kevin Washington 28:28
We added a ad that was really, really confusing, like right above reviews.

So there used to be an ad right here, and the problem with that ad was that it was massive.

So it was, imagine this ad that’s right here, only it was right above reviews, and it said,
get deals on best buy products.

Today, it’s been so it looked with all the Best Buy branding and whatnot.

And then you click on it, and it takes you to some random Capital One, like, sign up for
coupons experience.

And I just happened to notice that it was there, and I had to go ask somebody like, Hey, this
is super confusing.

And when I click on this, it seems like a terrible experience.

Like, who okayed this?

Like, did we test the impact of this?

Because I don’t remember seeing it.

And they were like, Oh no, we didn’t test that.

But we’re running a test very similar to that on the PLP, which is the when you go and search
the product list page, let’s see how that one’s doing.

And we lost a ton of money on that right?

Made the ad revenue, but clicks were way down and bounce rate because you were clicking that
and it was a non endemic ad that’s pushing you off of the platform.

That’s not what you want on your PLP and your PDP.

And so is that,

Matt Wright 31:11
is that, in that case, though, that like you are just creating the placement, like the real
estate that then can be filled by any ad.

Or do you, or is that ad an exclusive ad that’s already so it’s just the placement?

Kevin Washington 31:27
Is just the placement?

Yeah, okay, and so it was the potential for the risk of that ad being non endemic, because
they didn’t sell it correctly.

Matt Wright 31:38
When you say non endemic, does that referring to an endemic means like it’s or non endemic
means like it’s staying inside of the Best Buy domain platform, versus otherwise it’s actually
being redirected to a totally different brand page.

Yeah, right.

Kevin Washington 31:56
Okay, exactly that is exactly it non endemic would be.

It’s like, not sold by us.

So pushing us, pushing them outside of the Best Buy experience.

And so because it was a massive placement, and it was right here, it didn’t look like an ad
to people, and that that was just a risk, they had to be run.

So we actually it was during holiday and everything.

So we actually had to run an emergency AB test, where we left that as control.

Did a 5050, where we removed it to see what it would do, purchase conversion, because there
was this assumption that it had driven down revenue by like, you know, ridiculous amount of
money.

Turns out, no, in the short term, it wasn’t really driving a whole lot of that, a lot of that
money down.

But they still went ahead and made it a much smaller ad, because for the little bit that it
was causing chaos, making it endemic and smaller, got rid of all of the signals of losing
purchase conversion and bounce rate and all that.

31:38 — Ad placement risk + non-endemic ads (sending users off-site)

Matt Wright 33:05
This is like, probably quite a big question that probably has a difficult answer, but just
curious when you kind of have those dynamic parts of websites, maybe best buy in general, but
just more broadly, because I think of this one, especially when I’m reading things like a news
article or whatnot, that there’s clearly just like real estate there that any ad can fill it,
obviously, then these like different ad networks have signal on me, and they can drop,
hopefully, Like a relevant ad in that space.

But oftentimes I find like, it’s like, so distracting or like overwhelming that like that has
like, well, I’m there.

It’s okay, but I feel like the long term effect of that is like, oh, like, I’m slowly like,
just like, hey, like, like, I don’t enjoy being here.

I don’t enjoy reading this content.

Is there anything that you can think of, of, like, any way to control for that dynamic nature
of it?

Does it affect?

Like, it’s like, if, if a really crazy ad was running during like, a test period, that’s also
like, a super uncontrolled factor maybe, or maybe just doesn’t matter, because it’s all just
going to wash out anyways.

Like, do you have any like thoughts about that in terms of, like, is there a best practice
for that?

Is that is that just a concern?

Is that not relevant?

Kevin Washington 34:15
It’s one of those hard things. 

Because what I would say is best practices, you should have controls on a lot of those things, and especially like this particular example, is one where we were able to kind of advocate as
well.

We shouldn’t have been advocating as the experimentation team who’s got nothing to do with us, right?

Generally, right?

We run the experiments, what you do after the fact that’s on you.

I shouldn’t be telling the retail media team how to run their business.

But I did have to say, like there is a market difference between showing an endemic ad and non
endemic ad here.

And if you don’t, I know that the best practice is, if you didn’t sell the ad, just let
Google Ad Mark.

Place put an ad there.

Maybe that’s not the best practice that we should run on our extremely high value pages,
right?

Yeah, Pep, that’s rough.

Hey, homepage, sure.

Maybe right?

Some of these smaller whatever, hey, maybe just the fact that, like, the placements at the
very bottom of the screen, like, don’t really care what you put there, it’s all good.

We got a bunch of affiliate stuff here that looks super sketchy.

Sure, put the ad under that.

Make it as endemic, endemic, non indebted, whatever.

But when it’s right, you know, two components down from the price block, and I’m just trying
to get the information I need to go and convert.

You’re now putting an interruption in the middle of my buying experience that just doesn’t
make sense from a user experience perspective.

So, and this is back to the you know, metrics for people too.

We could run as many tests as you want that doesn’t pass the smell test on something that just
makes sense, right?

Show me as much data as you want as a consumer that should feel bad, and we can go do other
types of testing to determine whether or not that feels bad by actually talking to customers
and seeing how they feel about it.

And realistically, it’s hard to prove LTV at the end of the day.

It takes a long time, and by the time you’ve kind of proven out the LTV or the share of wallet
or anything else.

You’re three to six months down the line going, damn, we shouldn’t have done that last quarter.

So it’s a it, which is why a lot of we did a lot of that testing to try to figure out, what
are those drivers of that LTV over time and like, what makes the best customer?

Our best customer is somebody that has a, you know, let’s say I think it’s annual speed extra.

Let me not get into the actual numbers, but that was cheap.

There was, like, an annual spend number for example, right?

And so if that’s what I’m shooting for, for consumers in this cohort, and it was all cohort
by cohort based, then my assumption of their share of wallet is X on average.

And so can I see some of these behaviors start to drive towards that share of wallet or not?

And like, what categories are they spending in that type of thing?

33:14 — Controlling dynamic ads + LTV reality

Matt Wright 36:51
Yeah, fascinating.

Well, I just want to say I think this is like, loved it.

Love the conversation, especially, I love this idea of like, hey, like, kind of the jobs of websites are evolving.

The way that we can monetize them is evolving, I think, like you just kind of sharing this,
especially for something like Best Buy, of like, how they think through, they’ve got these
different tensions right between media networks and marketplace, and then just like the
baseline, what the site always did, I think, is super, super fascinating.

Thank you so much for doing this.

I really, really appreciate it.