5x increase in revenue for Shaw Academy with rapid experimentation program

On every Conversion project, experimentation starts on day one. During the kick-off meeting, the consultant presents the initial concepts – they then go into production almost immediately.

For Shaw Academy, these first two experiments delivered a +14% increase in sales (both at over 99% significance).

In one of those experiments, we optimized the messaging on the billing page, addressing the user’s primary objections.

In the control, users were unsure about entering their payment information. In the new variation we added copy that clarified when they’d be billed, and reassured them that they could cancel online at any time.

A simple experiment like this is perfect to kick off a new project. By changing UI rather than functionality, we can deploy the experiment rapidly. And because it’s focused on one of the users’ primary objections, it’s more likely to have a significant impact.

Testing bold ideas – with a safety net

The more experiments we ran, the more confidence Shaw Academy had in our methodology. In fact, at the time of writing, the program has a 38% win rate, with all experiments run to 99% statistical significance.

That high win rate meant Shaw Academy were prepared to try more ambitious experiments. After all, the true value of experimentation is not to test what you would have done anyway – it should allow you to take bigger risks, with a safety net, and reap bigger rewards.

In one experiment, we wanted to remind users when their free trial was about to end. That would mean we could reassure new customers during sign-up that we’d tell them before they got charged – but it also risked increasing cancellations.

Rightly, Shaw Academy were concerned about running this experiment. It could increase cancellations and – because free trials lasted 30 days – we’d have to wait over a month before we’d know about it. What’s more, Shaw Academy have a six-figure monthly spend on acquisition, so they’d risk losing a lot of cash as well as a lot of customers if the experiment failed.

So we defined an experiment to limit exposure, introducing a new challenger that reassured the user: “As a reminder, we’ll email you 3 days before”.

Given the importance of the experiment, we wanted to be confident that any successful result wasn’t a false positive. So we calculated the duration based on 99% statistical significance, and launched the experiment.

The impact on free trials was clear early in the experiment: the new challenger showed a significant increase.

But we needed to wait to see what would happen when this cohort of users received the reminder email. As it happens, it had a negligible impact on cancellations – so the positive increase in trials was carried through to sales as well.