Strategic testing: How to approach CRO for long-term profits and customer success

Strategy will make or break your experimentation program.

With no strategy in place, you risk running the wrong tests, in the wrong order, on the wrong goals.

But get the strategy right, and you’ll have an impactful and scalable experimentation framework.

What’s more, this framework can help you apply testing not just to your website, but across your entire organization. Testing then becomes the mindset for growth – not an occasional bolt-on to website marketing.

It enables you to test and optimize messaging, design, user experience – even your advertising, pricing and product.

There are key habits and indicators that suggest a testing program is more tactical than strategic. The table below compares the tactical vs. strategic approaches. Read through description for each to understand where on the spectrum between these two your current approach is lying.

With a tactical approach to testing, tests do not inform a well-integrated testing strategy, but exist in isolation. This means that when a test is over, you simply move onto your next planned test. As a result, your testing strategy look like this:

The diagram key:

• Tests marked as red = losers
• Tests marked as yellow = did not make any difference
• Tests marked as green = winners

Tactical approach to testing

It’s a random set of tests where some win, some lose, and none of them inform your subsequent steps.

In contrast, when you employ a strategic approach your testing looks like this:

Strategic approach to testing

In essence, levers are factors which the data shows may impact user behavior on the site. For example, the lever “card comparison” was based on research findings which showed that people find it difficult to compare credit cards on finance websites. As a result, they did not apply for any because they couldn’t decide which was best.

Levers inform branches of tests. Some tests win, some lose, but the tests are integrated, i.e. each test can inform subsequent tests based on its result.

For example, if you’re a pharmaceutical retailer and you found that delivery is an important factor when deciding whether to purchase oral contraceptives, then here’s what your first test could look like:

Original version of the page

Variation A of the same page

If your test won, then you could iterate that test idea. Was it the free delivery that mattered or was it the speed? Next step – two variations: “Free” vs. “Next Day”. If it was the speed, maybe we should introduce in-store delivery as well as next day delivery, and see if the extra expense is justified by extra demand. Then, we might test making it more prominent. Instead of showing it in the header, we could include as part of the page’s sub-headline.

This is how a strategic approach to testing forces you to amplify impact from your original test and uncover granular insights about your customers.

Once you know the intricate details about what motivates/prevents your customers from taking a desired action – you’re still not done!

The next stages you can (and should) go through are:

• Scale your impact, i.e. test the same concept on other pages. In the context of Lloyds Pharmacy this could mean reinforcing the same concept on other product pages (eg. if we tested it only on major brands, we could roll out the same test concept on the smaller brands, too) or we could test the same concept further down the funnel. For example, Lloyds Pharmacy could reinforce the same benefits when the visitor continues their order.

Applying insights to the check-out funnel

• Share your impact, i.e. apply the same concept to other areas of your business. If this concept resonated so well with your audience, let’s test including it in your PPC campaigns, meta description, and email marketing promo offers. If these work, there is sufficient evidence to then test these in your offline marketing, too!

Here’s the essence of it: You find a winning test idea and then you hammer it. To do so, follow this protocol.

If the test wins:

• Amplify (same concept, same page)
• Scale (same concept, other pages)
• Share (from acquisition to product)

To decide which levers are most powerful in changing your customers’ behavior, you need a broad view of your optimization program. For one of the clients we worked with we created the following (anonymised) table:

Lever performance breakdown

We tracked everything: the number of tests we ran around a certain lever, the win rate, the uplift that every test concept creates. We then segmented it based on different criteria: the step in the conversion funnel where it was executed, the acquisition channel, device type. This gives us a better idea of where we can scale the impact generated by our most successful tests. For example, we can see that “trust” had the highest win rate and a relatively large uplift, but we have not yet run many tests on our PPC traffic. Let’s scale it further!

It’s essential for a strategic testing program to maximise the value of its resources. The success of the program will be limited both by the volume of tests the website supports, as well as by internal resources like design and development time.

That’s why it’s essential to prioritize your tests effectively. It’s impossible to run every test we can think of – so we have to be selective and prioritize strategically.

That means planning the test roadmap by considering variables like:

• The value of the area being tested (eg the flow, page or element)
• The potential impact of the test
• The ease (design and build time, as well as sign-off) required to launch the test
• Ensuring that we’re learning about user behavior (eg by testing across a range of levers, rather than focusing heavily on one or two)
• Any risks associated with running the test

In short, we want to prioritize high-impact, high-ease tests on high-value areas:

By prioritizing tests based on impact and ease, you make sure that you don’t invest your time in complex, low impact tests.

If a test is complex but has a high potential impact, you should (whenever you can) try to prove the concept first. That means simplifying the execution of the test to a point where it becomes feasible to run – the “minimum viable test” – before progressing to more complex (and potentially more impactful) iterations.

Let’s consider an example.

Minimum viable test: Credit card industry

The research we conducted when analyzing the credit card industry showed that the fear of not being approved was the #1 reason preventing people from applying for credit cards.

Santander has a good example of a bad landing page. All the eligibility information is hidden under a huge block of text. Even if you find it, it’s generic, and there is no guidance on whether you, given your individual circumstances, would be approved.

Santander bank – eligibility criteria

To address this objection more effectively, Santander could build an eligibility checker similar to the one Barclays has:

Barclays bank – eligibility checker

However, it would require substantial time to build.

To understand if it is worth investing resources into this new tool, Santander could create a minimum viable test to first prove the concept. For example, they could add a new section at the top that would look similar to an eligibility checker, but upon clicking would still present the same generic information:

Example of a minimum viable test

The visitors still would not find out the information specific to their needs, but the important point is that Santander would be able to measure the % of people who click on this button. If they do, it’s worth developing the concept further – if they don’t, their resources can be better deployed elsewhere.

Every successful test should inform the overall testing strategy. But that can be a challenge if people on your team change and the knowledge of what worked might fade away. Without an effective knowledge base of tests, you’re facing two risks that can undermine your testing program:

1. Repeating previous tests: You might run similar tests again. At best, you may validate the previous result. At worst, you’ll waste resource by repeating a test – and potentially one that had a negative result.
2. Dismissing new concepts: A bigger risk is saying, “We already tested that”, without being able to show exactly what was tested and what the outcome was. As above, a test’s success is primarily down to the lever and the concept’s implementation. Dismissing the lever because of an unsuccessful earlier test is a huge risk.

Example of how Conversion.com documents its optimization program

To manage those risks more effectively, at minimum you must track:

• Hypothesis
• Creative execution (screenshots)
• Areas of focus
• Results (raw data)

But ideally you should also include external factors such as seasonality, competitors’ activity and market conditions. External factors can have an impact on your test results. For example, during December many ecommerce sites do not see their tests achieving statistical significance. This is due to the nature of demand. During peak periods, people care less about persuasive copy, layout and design – they just need to make a purchase. As a result, a well-crafted landing page may not perform any better or worse than the original, but once the peak period is over, clear differences start to emerge.

Here’s an example from Chris Goward’s book You Should Test That! None of the variations achieved statistical significance in December, but Variation C became a decent winner in January and conversion rate difference jumped from 12.7% to 30.6%.

Adapted from Chris Goward’s book You Should Test That!

When you approach your testing strategically, there are no such questions. You just go to your knowledge base of tests and analyze whether the test result was a result of the lever, the concept implementation, or potentially external factors (eg seasonality, a change in market conditions, or a change in the traffic profile).

This brings us to important point. If you’re a strategist, here’s how you should approach these losers.

If the test loses due to:

• Lever (the core theme of the test didn’t affect user behavior) = abandon
• Execution (the implementation – design, copy, functionality – didn’t affect user behavior) = retest (and reprioritize)
• External factor (eg seasonal or market conditions) = retest (and reprioritize)

(For a more in-depth discussion on why execution might fail your tests, read this article by Erin Weigel, Senior Designer at Booking.com)

Finally, at the heart of strategic testing is an alignment with the goals of your organization.

That means the KPI for your tests may not be the conversion rate from visitors to customers, but a broader goal like revenue, profit or lifetime value.

For example, if your goal is to increase revenue, you might break it down as:

Revenue = Traffic x Conversion Rate x Lifetime Customer Value (LCV)

It may be the case that simply putting up your prices will increase the LCV significantly, even if it decreases the conversion rate marginally. It can be a risk to test, but it’s often a simple test to run – there’s very little design and development work involved. This is especially true in some SaaS markets where customers are less likely to have an expectation around price, giving greater elasticity.

Example

This is exactly what Jason Cohen, the CEO of WP Engine, recommended to one of the companies at Capital Factory (the largest start-up incubator and accelerator in Austin). According to him, they doubled their prices and the effect on signups was minimal. As a result, the profits almost doubled. There you are – price inelastic demand.

So, should you also double your prices? This is what strategic testing can give you an answer to.

Risk profile

Transforming your organization means not only growing it, but also challenging its deep-seated assumptions.

For example, in SaaS this might mean re-thinking how you structure your pricing plans. Would customers be convinced to upgrade to higher-tier plans because they see more value in advanced features you offer (and should you thus structure your plans as in the image below)?

More features = higher price

Alternatively, you could test giving all features to everyone, regardless of the plan they’re on – then limit the volume of their usage instead. That way, every customer is able to experience the full benefits of the platform, and is more likely to use and engage with it, increasing their usage and subscription level:

Higher volume = higher price (same access to features for everyone)

(Or could you try and strike a balance between the two, or abandon the whole idea completely and simply charge a flat $99 fee the same way Basecamp does?)

Ultimately you need to maintain a healthy risk profile that’s appropriate for your organization and its testing maturity.

This means not only iterating your existing test ideas (= safer tests), but also testing completely new concepts and experimenting with radical changes. If you’re not nervous about even just a small percentage of your experiments – then you’re not being radical enough, and you risk not answering important strategic questions about your business and your customers.

Insight narratives

Ultimately, in order to transform the organization the research/data science team needs to align everyone on making data-based decisions. This means no more sitting together as a closed group that simply sends reports to the C-suite once a month, but becoming the core link between the C-suite and the business’s customers. This comes back to the point raised above: the impact in the form of new knowledge needs to be shared with the organization. Humans are hardwired for stories, not processing long spreadsheets. This is why storytelling backed by data – what we call insight narratives – is the most effective way to keep the data pumping through the veins of your organization and aligning everyone on the same vision.

Avinash Kaushik put it brilliantly (when he was interviewed at SES conference):

We need to take some of the dryness and connect it to real life when we present data. So, when people ask me what the metric bounce rate is, I very rarely say that it’s the percent of sessions with single pageviews. That does not communicate what they are! What I say is, they represent – from a customer perspective – an experience that is, “I came, I puked, I left”. You are never gonna forget that! You are never gonna forget that definition because of the way it was articulated.

I found that after years of trying to convince people, I’ve tried to get data to connect to real life. When a newspaper company wrote an email campaign and I analyzed it later, I basically said, “You had the 13 million one night stands with customers because you created no visitor loyalty”. Again, that was a way to make this data very REAL to them. Everyone knows what a one night stand is, and most of them were not great.

Avinash Kaushik Digital Marketing Evangelist at Google

As you can see, there are clear differences between tactical and strategic optimization programs.

It’s not to say that individual tactics won’t work – they can and do – but without a broader strategy to unite them, they’ll be limited in reach and impact. Sun Tzu, a Chinese military strategist, knew that the problem was not with the tactics themselves, but with the overall approach:

“Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat.”

With an effective strategy in place, it won’t just provide a framework for testing – it’ll allow you to test deep-seated assumptions in your organization.

And by doing that, you’ll be giving your organization a significant competitive advantage. While most companies are stuck testing granular changes to their websites, you’ll be testing changes that can radically shift your ability to acquire, convert and monetise traffic.