Combining UX Research and AB Testing methodologies at Whirlpool

Client Whirlpool
Industry Ecommerce

Throughout this case study, we will look at instances where mixing methods has helped to inform better experiments, uncovered nuances of different customer experiences and given Whirlpool critical insights into their customer that allow them a broader, greater understanding of their motivations.

Opportunity

Whirlpool Corporation is a brand committed to being one of the leading global appliance providers that consistently seeks to improve life at home. Whirlpool brings its shoppers a cutting edge, evolving digital experience built on a foundation of trust that drives purposeful innovations and solidifies the company’s position as a global leader.

An entrepreneurial brand at heart, Whirlpool is committed to innovation throughout its brand portfolio, and a robust experimentation program has allowed them to tap into new opportunities and create value throughout their digital shopping experience. 

When Whirlpool first partnered with Conversion, they had been running A/B and usability tests for about 6 months. Whirlpool was looking for an agency partner that had the expertise and experience to design and conduct research to learn about their users and an experimentation program that optimized the customer experience across all their brands.  

Our partnership sees a regular cadence of conducting A/B Tests and UX Research studies every month, tapping into the power of mixing methods and the unique way it enriches experimentation.

Whirlpool wanted to further develop their experimentation program by focusing on two key areas:

  1. An Initial Discovery Research Phase – Whirlpool was looking for a partner to help them conduct a research initiative and we conducted explore research to orient ourselves and the organization around their testable areas of opportunity via analytics audits, heuristic analysis, goal identification & mapping, discovery workshops, prioritization workshops, and creating a backlog of opportunities for experimentation.
  2. CRO Program Management Phase – Once these new opportunities were uncovered, Whirlpool was looking for a partner that would build and execute A/B and UX Research to drive meaningful improvement.

The circular relationship between qualitative and quantitative insights in these mixed methods has grown exponentially throughout our partnership, and continues to inform the testing strategy, user experience, and iterative innovation across Whirlpool’s brands. By harnessing Conversion’s proven methodology towards mixed methods experimentation, Whirlpool is now equipped with critical new customer insights and experimentation lift.  

What is Mixed Methods Experimentation?

Mixed Methods is the intricate process in which different methodologies are combined to generate different types of insights. Some of these techniques are quantitative in nature, like A/B testing, and some are qualitative such as certain User Experience Research (UX Research) methodologies. 

A/B testing and UX Research are highly complementary, and the combination of the two are on the cutting edge of marketing and customer acquisition. Utilizing mixed methods helps Whirlpool keep a constant pulse on the visitors of their site. There are a couple ways in particular that A/B Testing and UX Research work together pertaining to our engagement with Whirlpool:

  1. Understanding users and identifying hypotheses: UX Research helps us understand what’s most important, or what causes the biggest friction for prospects and customers. This allows us to prioritize which are likely to be the most impactful testing hypotheses.
  2. Avoiding stagnation through constant innovation: By utilizing the synergies created by UX Research and A/B testing, Whirlpool is able to continually be ideating, testing and iterating, supporting the business’ growth and keeping ahead of the curve.
Applying Mixed Methods Experimentation at Whirlpool

Mixing methodologies, that is combining both UX Research and AB testing, allows us to gain deeper and broader insights, validate findings across multiple data points, and ultimately have more impact for the business. There are 4 key ways that we apply UX Research to our experimentation program with Whirlpool:

  • To inform A/B Test Hypotheses
  • To explain A/B Test Results
  • To inform Design and Site Improvements
  • To understand user motivations, contexts, mental models and inform business decisions

Imagining a new Interstitial Experience

A key area of focus for Whirlpool has been on optimizing the interstitial experience, a modal which is shown to prompt users to select additional services and options. This Interstitial was shown to users when they select an appliance before they proceed to their cart, and was shown again after the cart as they move to checkout, if they hadn’t selected delivery options initially. 

An A/B test found that most users are ignoring or bypassing the interstitial experience. It also found that revenue per visitor (RPV) can be increased if users go straight to the cart when selecting an appliance and only see the interstitial once, when they are moving from cart to checkout. 

Following this test, Conversion reimagined the placement and look, feel, and function of the interstitial – beginning with a user research study to help inform that evolution. We conducted moderated interviews, recruiting individuals who were looking to purchase a major appliance online in the next 3 months. 

UX Research on the Control Interstitial Experience

Research Objectives:

To understand what points of the purchase journey shoppers expect and prefer additional services to be offered

  1. To gain insight into the user experience of the Whirlpool interstitial, and any associated friction points and areas of opportunity
  2. To inform the evolution of the interstitial experience to one with improved user engagement

The research uncovered important customer insights into what mattered most to shoppers looking for an appliance. When asked about additional services, the majority gave examples relating to Delivery & Installation and Warranty & Support. Delivery and Installation were seen as important, but preferences for extended warranties were mixed.

When users came across the interstitial before proceeding to their cart, they all commented that this was unexpected. When clicking ‘Add to cart’ on an appliance, users expected to proceed straight to their Cart. Many users were then confused on what the different options newly presented to them meant, which negatively impacted the perceived value for selections like the installation option:

The fact that the interstitial was unexpected, combined with the pop-up layout of this modal, helped to provide some explanation for the result observed in the AB test, where conversion was higher when the modal was shown after the cart, when users are proceeding to checkout.

UXR uncovered potential pain points, expectations users have on where additional services will be placed, and informed the layout and development of a new experience. Armed with critical new insights into what users were looking for, the UX Research helped inform the evolution of the interstitial experience, to be tested with an A/B test. As well as providing further insight into the previous AB test result, this UX Research study generated insights into user pain points and expectations which were applied to the next round of A/B testing, where we redesigned the interstitial experience to better align with the user needs and reduce the friction identified

Putting the Interstitial to the Test 

Control – a modal style interstitial served immediately after the add to cart

Redesigned Experience – Variations A and B – a full page stepped experience to “configure your appliance” served after “add to cart” in Variation A and after “proceed to checkout” in Variation B.

The redesigned interstitial was incredibly effective at increasing user engagement on desktop, however the interstitial was not as effective on mobile – this is likely because on a mobile device the experience was too similar to the control, and this change made the checkout experience feel longer for users.

Images for the Mobile Experience

Variation A

Variation A significantly reduced individuals from ignoring or skipping past the modal, as well as increasing delivery and warranty adds – but not the actual orders of these products. Both variants were significantly more likely to capture user attention and engagement – users were also significantly less likely to skip past the modal.

Variation B

Variation B was the clear winner with a 23.7% lift in orders with the warranty attached and a 21.4% lift in orders with paid delivery attached. An astounding result! Unsurprisingly, we discovered that users ignore the modal. A full screen takeover that presented the warranty and paid delivery option solves that problem. Variation B has such a significant impact at actual sales of these orders because we’re presenting users with these options at exactly the right time.

Understanding the Redesigned Interstitial Experience through UX Research

Run concurrently with the A/B Test another UXR study was commissioned that focused on understanding user experience with the newly redesigned interstitial, this time including both the desktop and mobile experience:

Research Objectives:

  1. To gain insight into the user experience of the redesigned interstitial, and any associated friction points and areas of opportunity
  2. To support the analysis of the concurrently running AB test, helping provide an explanation for the results observed
  3. To inform further testing and optimizations to the interstitial experience

To users, the redesigned interstitial felt like an expected step in their purchase journey

In the previous study on the control, users stated that the modal interstitial felt like a pop-up and as a result they instinctively wanted to click out of it.

In the redesigned flow, all users interacted with the related and recommended accessories sections. In the previous study, the positioning of the Related Accessories below the CTAs meant that several users didn’t initially notice this section.

One highlight from the study included users being positively surprised by the delivery options and pricing communicated in the interstitial, and even suggested this information to be more prominently displayed earlier in the journey:

In addition to this study identifying areas where the interstitial redesign had improved the user experience, it also helped to highlight areas which were still causing friction and would be the focus in the next stages of the interstitial evolution.

This work on refining the Interstitial experience is a great example of how complementary AB testing and UX Research are, and the power of using them together. Through UX Research we were able to help explain AB test results, inform the focus and designs for the next round of testing, and capture broader insights into how users think about additional services. This example also speaks to the additional value UX Research brings outside of experimentation; the Variant which completely redesigned the interstitial was not implemented by Whirlpool as the lift required far out wayed the lift required for the Variant which altered the placement of the interstitial but not the design, however the UXR findings were used to make updates to the current design to remove friction and improve the user experience.

Read Part 2 of our case study and learn how Whirlpool improved how users search and the effects a culture of experimentation has had at their company.

Want to work with us?

We work with everyone from small startups to some of the biggest brands in the world.

Get in touch and see how we can help you.

Contact us