Groomed for success: Building an experimentation program for a high growth business

Client Dollar Shave Club
Industry Ecommerce

Dollar Shave Club had recently invested in a top-of-the-range a/b testing tool. Conversion was hired to steer Dollar Shave Club’s experimentation program and generate a positive ROI from the tool. Within 3 months, we’d been able to launch enough winning experiments to cover the entire cost of the tool, as well as our annual service fee too. From there, we were able to expand the remit of the program, working closely with Dollar Shave Club’s internal team to ratchet up velocity and build an insight-driven program that informed decision making across every area of the business.

Opportunity

As a global leader in men’s grooming, Dollar Shave Club (DSC) understood the importance of innovation and experimentation. 

In fact, DSC’s leadership team was so bought into experimentation that it had decided to build an internal testing tool of its own – but this came with some unforeseen complications:

Due to the need for engineering resources associated with launching new tests, the DSC team had to choose between building new features or launching new experiments. This led to a guess-work prioritization queue that often resulted in larger initiatives being prioritized, while smaller, low-hanging test opportunities were filed away. 

Combined with other related issues, this meant that neither the program nor the tool were generating the impact that had initially been hoped for. 

DSC’s VP of operations, Tsega Ortega, believed that if the business was going to make a success of experimentation, it had to move away from its internal tool and invest in an external testing tool instead. 

Given the large investment that had already been made – plus the considerable cost of an external tool – this proposal met with much reluctance internally. 

The project had lots of visibility, the stakes were high, so it was absolutely imperative that Tsega was able to show a positive ROI from the tool right away. 

Unfortunately, it soon became apparent that the service provided by the testing tool itself was not going to be sufficient to make the programme successful – which is where we came in:

Conversion was hired with the short-term goal of driving immediate impact and achieving a positive ROI from the testing tool investment. 

Longer-term, we were tasked with fostering adoption of experimentation across the business, while using experimentation to generate significant customer insights that could be disseminated throughout the business to unearth new opportunities and accelerate growth. 

Overcoming inertia

As a customer-centric business keen to understand the evolving needs and desires of its users, Dollar Shave Club had already run some very strong foundational research that we were able to use to inform the first steps of our work. 

This research – a qualitative ethnographic in-home and  in-store study of male personal care shoppers – generated two core insights. 

  • There is lots of inertia in the male grooming shopping experience: men are unlikely to change brands unless they have a good reason to do so. 
  • This inertia carries over to buying habits too: most men purchase their grooming products off the shelf and they are reluctant to switch to an online subscription. 

This meant that not only did DSC have to convince their prospective customers to switch brands – a big ask in and of itself – they also had to persuade these customers to change their buying habits too. 

We knew this was going to be difficult – changing ingrained behaviors is always an uphill battle – but with strong expertise in behavioral science, we felt confident that we could drive the desired impact.

Solution

Since its inception in 2012, Dollar Shave Club has done a tremendous job of using viral marketing to build a playful, conversational, well-loved brand.  

Unfortunately, when we ran research to understand the emotional tone and brand feel of the website, we found that this playfulness was not translating to the digital experience at all. 

This was a big problem. 

Control: the original copy in the funnel was non-conversational, which didn’t match DSC’s brand tone of voice

Research from Harvard Business Review shows that customers who are more emotionally connected with a business convert better and spend significantly more money (52% on average).

harvard business school data on the impact of emotional resonance

We therefore hypothesized that if we could improve the emotional resonance of the funnel, we could increase the user’s emotional connection with the brand – and thereby increase the number of subscriptions we were generating. 

In one of our first experiments, we chose to tweak the copy on the first two steps of the funnel, switching from a clear but formal tone to a more conversational one. 

 

Variation: the new copy in the funnel was light and conversational, matching DSC’s brand tone of voice much more closely.

Worth emphasizing: this small copy test is the exact kind of test that would have previously been blocked. However, when we ran this version of the funnel against the original in an a/b test, it increased subscriptions by 5.24%. 

This was a powerful demonstration of the impact small tests can have. 

What’s more, the learning from this test – namely, that emotional resonance was a key conversion factor for the business – was quickly disseminated throughout the organization. Nowadays, all new experiences are written with strict style guidelines to ensure that the playful, conversational tone is always present. 

Fragmenting rewards

Another lever that our research indicated as being potentially fruitful was perceived value:  if we could find a way to increase the perceived value of the product, we stood a better chance of persuading users to change their brands and buying habits. 

When we ran a heuristic analysis of the subscription funnel, we noticed that most products were presented as a bundle. While this looked good – the images were aesthetically pleasing – it violated a well-known principle of behavioral design: 

Lump costs and fragment rewards. 

By lumping the costs together, you’re able to minimize their perceived importance (think ripping off a band-aid). By fragmenting rewards, i.e. splitting them out and emphasizing each one individually, you’re able to increase the amount of attention each individual reward is given – thereby increasing total perceived value. 

We saw this as an opportunity to increase the perceived value of DSC’s subscription offering, so we decided to redesign certain page elements: specifically, we separated out each of the products individually and included a bullet-point inventory of items included as part of each individual product. 

In the control (left screenshot), products and features were lumped together. In the variation (right screenshot), each product (and its features) were split out and shown separately.

We then ran this new version of the page against the original version in an a/b test, and it increased the number of subscriptions by 11.2%.

This was a huge result in and of itself, but when we analyzed the data for this test, we realized there was plenty more room for improvement…

Iterating for impact

Although the conversion rate had increased significantly, we also found that many users were using the ‘change your selections?’ link to go back and forth between different pages. This is known as ‘pogosticking’ and it’s usually a bad sign from a UX perspective.

While subscriptions were up, further analysis revealed that a higher than expected percentage of users were ‘pogosticking’.

In this instance, the pogosticking suggested that users were going back and forth to see how different products affected the price. 

To remedy this issue, we iterated on the original test by creating a new variation that clarified the bundling discount right from the start – ‘You qualify for a 15% Handsome discount’ (also, notice the playfully named discount, which squares nicely with DSC’s brand). 

We also wanted to see how removing the default checkboxes impacted the user experience and aided comprehension of the offer. We therefore created a second variation where we deselected some of the originally selected products. 

We then ran both of these variants against the previous winner in an a/b/c test. 

In the first variation (screenshot in the middle), we added copy to help clarify the bundling discount. In the second variation (right screenshot), in addition to clarifying the offer we also deselected default checkboxes.

As hoped for, the first variation increased the conversion rate by an additional 1.5%. Even more impressive, the second variation – with the deselected boxes – increased the number of subscriptions by 6.8%. 

This meant that across the two experiments – the original debundling test and its iteration – we had been able to increase the subscription conversion rate by more than 17%. 

Result

In the initial 3 months of our engagement, we were able to generate enough additional revenue to cover the costs of our services AND the testing tool for an entire year. 

Tsega’s conviction had proven right – a positive ROI was assured. 

With the initial goal comfortably hit, we were subsequently tasked with expanding the scope of our work far beyond that of the initial engagement. 

Part of this broader remit involved auditing DSC’s existing experimentation capability. Here, we used our PACET framework to identify any bottlenecks and blockers in DSC’s existing experimentation setup. From there, we then worked closely with DSC’s internal team to remedy any deficits and provide upskilling where necessary. 

Another element of this expanded remit involved extending  the impact of experimentation across the entire business. Part of this was simply about using experimentation to optimize other key growth metrics beyond subscription – like monetization and retention. Another was about ensuring that experimentation informed decision-making across all business units – from marketing to product, engineering to design. 

All in all, the program has been a huge success. 

We’ve built a strong consultancy partnership that the DSC team can trust. From fully understanding DSC’s needs to building effective experimentation strategies that drive impactful customer insights, we’ve become an essential part of the Dollar Shave Club team. 

In the initial 3 months of our engagement, we were able to generate enough additional revenue to cover the costs of our services AND the testing tool for an entire year. 

Tsega’s conviction had proven right – a positive ROI was assured. 

With the initial goal comfortably hit, we were subsequently tasked with expanding the scope of our work far beyond that of the initial engagement. 

Part of this broader remit involved auditing DSC’s existing experimentation capability. Here, we used our PACET framework to identify any bottlenecks and blockers in DSC’s existing experimentation setup. From there, we then worked closely with DSC’s internal team to remedy any deficits and provide upskilling where necessary. 

Another element of this expanded remit involved extending  the impact of experimentation across the entire business. Part of this was simply about using experimentation to optimize other key growth metrics beyond subscription – like monetization and retention. Another was about ensuring that experimentation informed decision-making across all business units – from marketing to product, engineering to design. 

All in all, the program has been a huge success. 

We’ve built a strong consultancy partnership that the DSC team can trust. From fully understanding DSC’s needs to building effective experimentation strategies that drive impactful customer insights, we’ve become an essential part of the Dollar Shave Club team. 

 

Looking to learn more about our collaboration with Dollar Shave Club? You can watch our webinar with the DSC team at the link below

Watch now

Want to work with us?

We work with everyone from small startups to some of the biggest brands in the world.

Get in touch and see how we can help you.

Contact us