Improving how users search & a culture of experimentation at Whirlpool

Interested in learning about what Mixed Method Experimentation is, and how it helped Whirlpool imagine a new interstitial experience? Read Part 1 of our case study!

Whirlpool collects a steady stream of user feedback via on-site polls and intercepts. One of the most common points of feedback and frustration from users was the search experience – a key opportunity for the team to seek improvements, but also a complex task given the many components that power the on-site search experience. Whirlpool had the challenge of trying to completely overhaul their search experience including everything from the navigation search bar and auto-suggest functionality, the experience of the search results page itself, and the underlying data sources and search algorithms that power the results. The team had proposals from a number of different options for search providers and search experiences that were difficult to evaluate on hypotheticals alone – so we put them to the test!

Search Bar and Search Results Control and Variations

Search Engagement and Paths and E-Commerce Funnel

Across all variations we saw how the experiences moved more visitors straight to PDP and had variable success in RPV, the primary metrics for this experiment.

In order to better contextualize the results and truly understand the impact of the A/B tests we sought to conduct some research to undertake the following objectives:

1. To explore different Search experiences, and capture related user preferences and friction points
2. To gather additional insights to compliment the A/B tests, and inform next steps

UXR for Search Bar and Search Result Page:

For the UXR portion of this work, we conducted unmoderated interviews with 12 participants, who were all looking to purchase a food mixer in the next 3 months and considering KitchenAid. We split this group into 3 cohorts, with each cohort experiencing and giving feedback on a preview link for one of the Variants. This approach meant we were able to analyze each Variant in isolation, and understand the different areas of delight and friction for each, as well as highlight trends that came out more broadly across the experiences.

When interacting with the Search bar within the navigation, users responded particularly well to Variants which included product images alongside text suggestions. They commented on the uniqueness of this experience compared to other e-commerce sites they were familiar with, and found it helped tangibilize the products and aided them in visualizing the different categories they were browsing. In addition, having information such as price and review ratings was helpful; in Variants without this information users were more reluctant to click on a product suggestion, as they didn’t feel they had enough information to move forward.

In addition to instructing users to interact with the Search bar, we also gathered feedback on the Search Results page, giving users a set of tasks and questions to work through. For this page, there were some smaller differences between the Variants, but what was more interesting were the bigger pain points that were highlighted consistently across the experiences.These included the unintuitive order that products were shown, frustration relating to the format of the price filter and difficulty in locating documents for Owners.

Bringing it all together for ‘Search Bar’ 

The A/B tests showed a definitive impact on user behaviour which was taken into account alongside other factors for Whirlpool such as cost and technical components. The UXR was useful both to understand user experience and the related delighters/friction across the three different variants, as well as to capture broader insights around search which could then be incorporated into the design regardless of which variation they landed on.

The Search bar experiment speaks to the ability of UXR to help explain A/B test results, as well as to inform design and site improvements. We were able to understand the user experience/attitudes of the different variations, to use in combination with the behavioural impact observed in the A/B test. The UX Research conducted identified some insights which we couldn’t have gotten from an A/B test alone, and which can be applied regardless of the competitor selected.

Armed with qualitative and quantitative insights on each of the three variations that were tested, Whirlpool was now in a position to be able to make a determination on which of these variations they may choose to move forward with. By using UXR to help determine the strengths and potential tradeoffs for each design, Whirlpool would be in a position to make a more confident decision on how the search experience should be like for their users.

Exploring a Culture of Experimentation at Whirlpool

We interviewed our Whirlpool partners Varun Suri, Experimentation & UX Strategy Design Leader, and Amberdeep S. Aurora, D2C eCommerce Product Management Lead to find out more about the culture of experimentation at Whirlpool:

1. How has a culture of experimentation informed and influenced decision making at Whirlpool, what are some tangible and intangible benefits?
   
   Amberdeep S Aurora: Mixed Methods has helped us understand the Why in addition to the What. So in cases where we’ve been puzzled by the results of an A/B test, the Mixed Methods approach has helped us understand the why behind it. Also, it’s helped us inform strategic direction on projects.
   
   Varun Suri: The holistic approach of mixed methods has removed any biases or challenges that we used to face while implementing or prioritizing these winning ideas that would drive conversion and a better user experience; equipped with the right data from a quantitative and qualitative side was enough to convince the leadership and move that idea for development & execution. Just having the winning numbers was not enough as we wanted to hear from our users directly on what are the friction points or unmet needs. The combination of AB Testing & UXR studies has greatly improved our understanding of our users and provided a winning design which is frictionless and intuitive.

2. What difference has a testing culture had on the employees and organization as a whole?

   Amberdeep S Aurora: Building a test & learn culture is a work in progress, but Conversion’s partnership has definitely helped move us ahead, and helped us move away from decisions based on opinions, to decisions based on data.

   Varun Suri: Testing culture is still relatively new to our ever growing D2C team, traditionally we have heavily relied on Analytics / other tools to gain insights as to how a feature is doing and that used to happen all pre & post launch. This testing culture has brought about a change where we can test and learn before we put our development resources (which are always a challenge to get) into developing a new feature. This has unlocked the following:

   • Testing ideas before requesting it to be developed which are driven from Business needs+roadmaps / VOC surveys / Industry best practices
   • Implementing features or experiences without fear of conversion impacts, through non-inferiority type testing to unlock other business goals besides conversion like improvements to SEO
   • Brought our users closer to our business units and bring that empathy for them

   The leadership & business teams are warming up to the ‘test and learn’ culture and reaching out to us with hypotheses+ideas which would help us build the optimal experience which helps unlock our D2C potential.

3. Looking to the future, what ways do you see experimentation and research evolving inside of Whirlpool?

   Varun Suri: Currently this is being done in pockets of the organization. We have made strides with experimentation in the digital part of our organization and hopefully in the future can extend this to the broader organization that builds physical products.

By harnessing the power of mixed methods, Whirlpool has been able to tap into critical customer insights and create new customer experiences that help them deliver the right experience to the right person at the right time.

In combining UX Research with A/B / Non-inferiority / multivariate testing, Whirlpool is well situated to continue to grow their experimentation program and iterate on the insights gained with the experimentation partnership we built together. Learn more about how mixed methods can help your organization today – reach out to talk to one of our consultants.