A/B Testing During Black Friday Promotions

Black Friday, and its counterpart Cyber Monday, are some of the largest consumer transaction days for the American economy. Shoppers come out in droves to either complete their holiday shopping, upgrade large ticket appliances/electronics, or just take advantage of great deals and promotions. Those of us who have ventured to participate in Black Friday or Cyber Monday have experienced long lineups in every part of the retail process, and even worse: slow and buggy websites.

However, many of us are so motivated by the fear of missing out on a great deal that we are willing to put up with seemingly anything to check off our holiday shopping list! Let’s take a closer look at why this happens, and how companies can build a testing and shopping experience around it to better understand their shoppers and lift conversion rates throughout the year.

The BJ Fogg model can help companies understand persuasive design and user behavior. The three parts of this model are “Motivation” up the y axis, “Ability” across the x , and an action line curve in the model as illustrated below:

What the BFM tells us is that:

1. If you have low motivation to do something, but that thing is “easy to do”, there is a strong likelihood that you would do it
2. If something is ”hard to do”, but you are highly motivated to do it then you are also likely to put up with the difficulty and still do it

What we help our partners do with experimentation, is take the baseline point which sits somewhere in the space along the axis of the Fogg model, and move the desired action above the action line, it is here where the user is most likely to convert and complete their checkout journey. If the desired action remains below the action line, then the shopper is much less likely to convert.

How do we do this? We can think of the optimizations we make to a website as focused on improving a user’s motivation to convert, ability to convert, or both.

Hypotheses aimed at improving a user’s ability to convert may be focused around reducing steps in the funnel or process, improving product discoverability, or increasing ease of use on the site.

Alternatively, hypotheses aimed at improving a user’s motivation to convert may be focused around improving the value perception of the product, presence and depth of product information, or – you guessed it – sales (an urgency tactic).

Events like Black Friday are unique in that they cause a dramatic shake-up in both user motivation & ability, creating an opportunity to convert individuals who may otherwise be below the action line.

We can see below the substantial shift in the model, with shoppers becoming highly motivated during the Black Friday period, however the ability to shop during this period also decreases. This can be because of long lineups at crowded stores, or websites experiencing an influx of traffic and rapidly fluctuating stock levels that makes them slow, unresponsive, or unreliable.

The focus here should be on increasing the shoppers ability to purchase and not their motivation, as their motivation is already at very high levels. We can see in the graphic below that moving a user along the ability axis is most likely to make the shopper convert, while making equivalent improvements in motivation will still likely not yield any conversion. We are experiencing diminished returns on motivation.

Black Friday and Cyber Monday can sometimes see an expected 173%-380% increase in sales. A massive increase no retailer can afford to ignore! As such, many retailers with an e-commerce presence will have developed plans for Black Friday far in advance, whether it be items they wish to feature or customized items stocked directly for the event.

As a result, users are flocking to shop online with a heightened motivation to purchase. They likely have a product (or a list of products) they’re looking to shop for and while price will be the key driver of their purchase decision, the ability to purchase will also play an important role.

Historically, some companies might be hesitant to test during Black Friday — the reason being they don’t want to “break something”, and consequently risk losing customers because of a sub-optimal process. Some companies will institute a “code freeze” on Black Friday/Cyber Monday, where no new code is to be added to a site at all for a period of 1-2 weeks around the actual date in an attempt to mitigate the risk of any unintentional website performance issues caused by new code releases.

With more mature experimentation programs, we have seen companies capitalize on Black Friday to run tests that help them gain critical new insights and maximize the impact of their site performance which pays dividends over the long term.

While it’s entirely reasonable to acknowledge the potential risk of testing through the Black Friday period as we have already mentioned, there also exists a large opportunity cost if choosing not to test during this time.

By not experimenting on this surge of traffic and conversions, you’re not only missing a high volume of users to test on, you’re also essentially making a bold statement that your website as it exists today is the most optimal and performant version of your site – which as optimizers we know is never the case!

While there is a risk of running a test and sending traffic to a sub optimal Variation A or Variation B, there is also the risk that your control IS the sub optimal experience and you’re missing out on the big gains throughout the year by not capitalizing on the opportunity of Black Friday testing.

So how do we mitigate the risk? Companies can focus on testing methodologies like a Multi Armed Bandit (MAB) through this time period. A MAB automatically diverts traffic to the better performing variation in your experiment. You gain fewer insights, because traffic is automatically diverted to the best performing variation, however revenue generation is maximized and risk is mitigated. If a variation (or your control!) is underperforming, the algorithm figures it out and reduces the traffic exposed to that version during the test period. Sounds pretty optimal for a critical time like Black Friday doesn’t it?

The MAB strategy may be the better option for a momentary period of high externality like Black Friday, as the question still remains how representative shopper behavior is during this time. As we demonstrated above, users are likely far more motivated in this period than on average. As a result, the tactics and variations that perform best during Black Friday may not be the best tactics during normal periods. For this reason, you may choose to revalidate some winners from MAB testing in a different period after the holidays to uncover even more insight about the differences in your shopper behaviors during these periods. The difference in performance post-externality could be a big unlock for what you do next year.

If you choose not to test during the Black Friday/Cyber Monday shopping period, you can still take this time to learn about users through qualitative research. Interviews can be conducted at a later date with some of the shoppers that chose to visit your site during the Black Friday/Cyber Monday period for further external insights.

Customer insights can help you develop future tests and even help inform brand strategy. Many shoppers will be newly introduced to your site during the uptick in visits during Black Friday and Cyber Monday. Conducting experiments and testing during that time period can help you gain new insights about these users with the potential of turning them into repeat customers.

Ready to learn more about how Conversion can help you with your CRO efforts, including testing during the holidays? Get in touch with our team today.