• Within our data-driven industry, many gravitate towards heavily relying on quantitative data to guide and inform experimentation. With Google Analytics and the metrics it measures (e.g. conversion rate, bounce rate and exit rates) often dominating our focus, it means many undervalue or forget about the other insights we can run.

    Numbers, rates and statistics are great for finding out what’s happening on your site and where the opportunities for testing lie. However, what some people still don’t understand is that quantitative insights can only take us so far within conversion rate optimization. Understanding where to target our tests for the best impact is necessary but it does not provide insight into what our tests should entail. This is where qualitative research takes center stage.

    Qualitative research provides us with insight into the “why” behind quantitative research. It provides a deeper understanding into your visitors and customers, which is vital to understanding why they behave and engage with your site in a particular way. Conducting qualitative research is ideal for discovery and exploration and a great way of generating insights which can be used to guide future experimentation.

    In this series, we will cover the qualitative insights that we run for our clients at Conversion.com including case studies of when qualitative research has helped with our tests and some of our best practices and tips!

    • Contents

  • On-site Surveys

    By on-site surveys we are referring to targeted pop-up surveys which appear on websites asking either one, or a series of questions to users to gather insights. Hotjar and Qualaroo are two of our favourite data collection tools for this specific insight.

    On-site surveys let you target questions to any visitor, or subset of visitors, to any page within your website. These surveys can be prompted to appear to your visitors in a number of ways; time elapsed, specific user behavior such as a clicked element or exit intent, or custom triggers using Javascript. Understanding the behavior and intent of website visitors allows us to more effectively interpret motivations and barriers that they may face with your site. These insights can then guide our data-driven tests which aim to emphasise the motivations whilst eliminating the barriers.

    On-site surveys have many benefits such as being non-intrusive, immediate in collecting data and they are anonymous which allows for higher ‘ecological’ validity of responses. But most importantly, they have the benefit of gaining real-time feedback from users ‘in the moment’ while they are engaging with your site.

    Don’t underestimate the power of an exit survey. An exit survey is triggered when a user shows intent to leave a website, for example, when a user moves their cursor towards the top of the page. Exit surveys are the best non-intrusive qualitative method that provide crucial insights into why visitors are not converting or why your website may have a high exit rate. They often outperform other website surveys in terms of response rates because they minimise the annoyance a survey gives to a user, especially when they were already planning on leaving the site.

    But what questions should you be asking in these on-site surveys? Well, that really depends on what you want to get out of this piece of insight. Below are a few examples of the types of questions you can ask:

    • Investigating user intent and bounce rate
      • Why did you come to this site today?
      • What were you hoping to find on this page?
      • Did this page contain the information you were looking for?
    • Understanding usability challenges
      • Were you able to complete your task today? (If yes, why? If no, why not?)
      • Is there anything on the site that doesn’t work the way you expected it to?
    • Uncover missing content
      • What additional information would you like to see on this page?
      • Did this article answer your question?
    • Identify potential motivations
      • What persuaded you to purchase from us?
      • What convinced you to use us rather than a competitor?
      • What was the one thing that influenced you to complete your task/purchase?
    • Identify potential barriers
      • What is preventing you from completing your task?
      • What is stopping you from completing your checkout?  
      • What concerns do you have about purchasing from us?

    When launching a survey, it may be difficult to know how long to run it for or how many responses you actually need. In reality, large sample sizes are important when collecting data, however we are more concerned with gaining in-depth understanding into your users, while looking for ideas and inspiration. Therefore we look for thematic saturation, the idea that the data is providing us with no new significant information, instead of large sample sizes. For more information about the sample size required to running an on-site survey and how many responses are necessary, check out our article about on-site surveys and thematic saturation.

    At Conversion.com we ensure we are continuously collecting both qualitative and quantitative insights on our clients. On-site surveys are just one of these insights which help to guide and backup our data-driven hypotheses.

    An example of when on-site surveys have guided tests to add additional revenue to our clients is with one company within the online pharmacy industry. Our on-site survey asked users what stopped them from choosing a particular product when they were at a specific stage of their user journey. Insights demonstrated that users found it difficult to choose products by themselves with no assistance or help. This informed a test we ran on these specific pages which implemented signposting to particular products through recommendations. We believed that by including elements that could aid users at the product selection stage, it would make it easier for users to select a product, thus eliminating the barrier we found via our on-site survey. Making this change caused an uplift of 3.48% in completed purchases for new users.  

    Look out for our next article in this series where we discuss the importance of competitor analysis.