Combining UX Research and AB Testing methodologies at Whirlpool

Whirlpool Corporation is a brand committed to being one of the leading global appliance providers that consistently seeks to improve life at home. Whirlpool brings its shoppers a cutting edge, evolving digital experience built on a foundation of trust that drives purposeful innovations and solidifies the company’s position as a global leader.

An entrepreneurial brand at heart, Whirlpool is committed to innovation throughout its brand portfolio, and a robust experimentation program has allowed them to tap into new opportunities and create value throughout their digital shopping experience. 

When Whirlpool first partnered with Conversion, they had been running A/B and usability tests for about 6 months. Whirlpool was looking for an agency partner that had the expertise and experience to design and conduct research to learn about their users and an experimentation program that optimized the customer experience across all their brands.  

Our partnership sees a regular cadence of conducting A/B Tests and UX Research studies every month, tapping into the power of mixing methods and the unique way it enriches experimentation.

Whirlpool wanted to further develop their experimentation program by focusing on two key areas:

1. An Initial Discovery Research Phase – Whirlpool was looking for a partner to help them conduct a research initiative and we conducted explore research to orient ourselves and the organization around their testable areas of opportunity via analytics audits, heuristic analysis, goal identification & mapping, discovery workshops, prioritization workshops, and creating a backlog of opportunities for experimentation.
2. CRO Program Management Phase – Once these new opportunities were uncovered, Whirlpool was looking for a partner that would build and execute A/B and UX Research to drive meaningful improvement.

The circular relationship between qualitative and quantitative insights in these mixed methods has grown exponentially throughout our partnership, and continues to inform the testing strategy, user experience, and iterative innovation across Whirlpool’s brands. By harnessing Conversion’s proven methodology towards mixed methods experimentation, Whirlpool is now equipped with critical new customer insights and experimentation lift.  

What is Mixed Methods Experimentation?

Mixed Methods is the intricate process in which different methodologies are combined to generate different types of insights. Some of these techniques are quantitative in nature, like A/B testing, and some are qualitative such as certain User Experience Research (UX Research) methodologies. 

A/B testing and UX Research are highly complementary, and the combination of the two are on the cutting edge of marketing and customer acquisition. Utilizing mixed methods helps Whirlpool keep a constant pulse on the visitors of their site. There are a couple ways in particular that A/B Testing and UX Research work together pertaining to our engagement with Whirlpool:

1. Understanding users and identifying hypotheses: UX Research helps us understand what’s most important, or what causes the biggest friction for prospects and customers. This allows us to prioritize which are likely to be the most impactful testing hypotheses.
2. Avoiding stagnation through constant innovation: By utilizing the synergies created by UX Research and A/B testing, Whirlpool is able to continually be ideating, testing and iterating, supporting the business’ growth and keeping ahead of the curve.

Applying Mixed Methods Experimentation at Whirlpool

Mixing methodologies, that is combining both UX Research and AB testing, allows us to gain deeper and broader insights, validate findings across multiple data points, and ultimately have more impact for the business. There are 4 key ways that we apply UX Research to our experimentation program with Whirlpool:

• To inform A/B Test Hypotheses
• To explain A/B Test Results
• To inform Design and Site Improvements
• To understand user motivations, contexts, mental models and inform business decisions

Control – a modal style interstitial served immediately after the add to cart

Redesigned Experience – Variations A and B – a full page stepped experience to “configure your appliance” served after “add to cart” in Variation A and after “proceed to checkout” in Variation B.

The redesigned interstitial was incredibly effective at increasing user engagement on desktop, however the interstitial was not as effective on mobile – this is likely because on a mobile device the experience was too similar to the control, and this change made the checkout experience feel longer for users.

Images for the Mobile Experience

Variation A

Variation A significantly reduced individuals from ignoring or skipping past the modal, as well as increasing delivery and warranty adds – but not the actual orders of these products. Both variants were significantly more likely to capture user attention and engagement – users were also significantly less likely to skip past the modal.

Variation B

Variation B was the clear winner with a 23.7% lift in orders with the warranty attached and a 21.4% lift in orders with paid delivery attached. An astounding result! Unsurprisingly, we discovered that users ignore the modal. A full screen takeover that presented the warranty and paid delivery option solves that problem. Variation B has such a significant impact at actual sales of these orders because we’re presenting users with these options at exactly the right time.

Understanding the Redesigned Interstitial Experience through UX Research

Run concurrently with the A/B Test another UXR study was commissioned that focused on understanding user experience with the newly redesigned interstitial, this time including both the desktop and mobile experience:

Research Objectives:

1. To gain insight into the user experience of the redesigned interstitial, and any associated friction points and areas of opportunity
2. To support the analysis of the concurrently running AB test, helping provide an explanation for the results observed
3. To inform further testing and optimizations to the interstitial experience

To users, the redesigned interstitial felt like an expected step in their purchase journey

In the previous study on the control, users stated that the modal interstitial felt like a pop-up and as a result they instinctively wanted to click out of it.

In the redesigned flow, all users interacted with the related and recommended accessories sections. In the previous study, the positioning of the Related Accessories below the CTAs meant that several users didn’t initially notice this section.

One highlight from the study included users being positively surprised by the delivery options and pricing communicated in the interstitial, and even suggested this information to be more prominently displayed earlier in the journey:

In addition to this study identifying areas where the interstitial redesign had improved the user experience, it also helped to highlight areas which were still causing friction and would be the focus in the next stages of the interstitial evolution.

This work on refining the Interstitial experience is a great example of how complementary AB testing and UX Research are, and the power of using them together. Through UX Research we were able to help explain AB test results, inform the focus and designs for the next round of testing, and capture broader insights into how users think about additional services. This example also speaks to the additional value UX Research brings outside of experimentation; the Variant which completely redesigned the interstitial was not implemented by Whirlpool as the lift required far outweighed the lift required for the Variant which altered the placement of the interstitial but not the design, however the UXR findings were used to make updates to the current design to remove friction and improve the user experience.

Read Part 2 of our case study and learn how Whirlpool improved how users search and the effects a culture of experimentation has had at their company.