[Introduction]
Host (Matt):
Excited to have you on for another bonus episode of the Conversion Podcast today. This one’s very cool because I feel like it’s a classic series of experiments. So, once again, please set the scene for us.

[Setting the Scene]
Guest (James):
Cool. So today, we’re talking about a few experiments we ran with a former client of ours. It was focused primarily on how we can use video to improve product comprehension. The idea was to use video to enhance understanding of a product from the very beginning. What they sell is actually a cell phone signal booster—a technically difficult product that not everyone is familiar with. So, comprehension plays a big role in the customer journey, especially in terms of how the product works.

What happened is they had this landing page that was performing quite well from a specific campaign they were running. They came to us and said, “Hey, this is performing so great, what if we used components of this as our homepage?” They felt people were engaging well with it, and it was converting fairly well. So, we did exactly that. We changed a handful of things, like different headlines and imagery.

One of the key changes was making the “How It Works” video very prominent on the homepage. We placed it as a secondary call-to-action (CTA) next to the primary CTA in the hero section. The prospect was doing a great job of explaining the product to people, so we thought maybe that’s why it was performing so well on these paid landing pages.

So, we tested it out. Of course, there were a lot of changes, so it wasn’t your typical isolated experiment. But how did it perform?

[The Initial Experiment: Video on the Homepage]
Host (Matt):
What were the results?

Guest (James):
At first, it went kind of how we expected. We made the video more prominent right in the hero section, and we saw a 150% increase in video views. To clarify, this was an increase in views because previously, there was a small place where you could watch the video in the bottom right corner of the hero area. By making it more visible, we saw a big boost in views—150%. Great, people are watching the video and understanding how the product works.

But the asterisk here is that it actually resulted in a greater-than-10% decrease in sales. So, despite the huge increase in video views, we saw a drop in sales, which was the opposite of what we wanted. This led to a bit of a “facepalm” moment because our goal was to drive more sales, not just more video views.

But, after analyzing the results, we realized something interesting. The 150% increase in video views was negatively correlated with sales. So, we began to ask ourselves, “Where else might this be happening on our site?” We realized that this same video was also featured on almost all of the primary product pages—right in the center, overlaid on the product image. Was that actually hurting us too?

[Adjusting the Strategy: Moving Video and Testing a New Approach]
Host (Matt):
What did you do next?

Guest (James):
We took what we learned from the homepage test and decided to do almost the opposite strategy. Instead of featuring the video prominently, we moved it down to the image carousel. This way, people who wanted to watch it could still find it, but it wasn’t front and center. We replaced the video with a static value proposition, something like “Up to 32x stronger cell phone signal.” Then, we tested this against the original setup, and, once again, we saw a very different result.

We got a 30% decrease in video views, which was a significant change in user behavior—fewer people were watching the video. But this time, the correlation held in the opposite direction. The 30% drop in video views led to a 35% increase in sales. So, what was a loss on the homepage turned into a win elsewhere when we applied those learnings.

[Analyzing the Results and Learning from the Experiment]
Host (Matt):
That’s amazing. It’s such a good example of how testing and iteration work. You had to go through the trial and error to uncover the right strategy, and now you have a more effective approach.

Guest (James):
Exactly. We learned that improving product comprehension is still important. For a technically complex product like this, customers still had a lot of questions. We saw queries coming in through customer service, “Does this work with all carriers?” and so on. We used this data to adjust our approach, incorporating more static content to answer those questions clearly.

We also experimented with a step-by-step guide on how to set up the product, which led to a 6% increase in sales. So, even though the video didn’t work in the way we hoped, we still achieved a significant improvement in comprehension and sales using a different method.

[The Power of Iteration: Compounding Results]
Host (Matt):
That’s so fascinating—how iterating and analyzing the results turned what was initially a loss into multiple wins.

Guest (James):
Yeah, exactly. It’s a classic example of how powerful iteration is. Even if something is losing, you can still learn from it and adjust your strategy. By figuring out the elasticity and understanding the results, you can adapt and use those insights to your advantage. The effects can really start compounding over time.

Every optimizer should think about running experiments like this—because you can apply this process to virtually any situation. It’s a strategy that works across the board.

[Closing Remarks]
Host (Matt):
Definitely. Thank you so much for sharing this example.

Guest (James):
Thank you!