5 experimentation industry predictions for 2024

As we begin rolling out our plans for 2024, we’ve been doing a lot of thinking about what’s likely to come next for the CRO/experimentation industry.

Will AI finally make true 1-to-1 personalization possible?

How will economic pressures affect accountability and decision making?

How is new legislation likely to impact our space?

In this post, we share five of our predictions for the industry in 2024.

1. Increased budget scrutiny = more experiment-backed decision making

In today’s struggling global economy, many businesses are looking to make cuts and minimize waste. Such cost cutting means that budgets are under huge scrutiny, which is likely to raise the evidence threshold for certain types of budget-related decisions.

For example, should we invest in branding or performance marketing? Should we hold our ad spend steady during the downturn or should we tighten our belts while CACs are high? Which pricing model works best for our business?

No longer will the gut feelings of HIPPOs – or herds of HIPPOs – be sufficient to answer these questions. Instead, we believe that business leaders are going to seek out high-quality, reliable evidence to guide their decisions during these challenging times…and what evidence is higher quality and more reliable than experiment data (randomised controlled trials?

Hierarchy of evidence

We therefore believe – and are hopeful – that economic pressures are going to lead to a rise in experiment-backed decision making.

2. More budget scrutiny will force a more honest accounting of program results (decay effects, holdout groups, etc.)

Tying in with the point above, we’ve recently found that many of our clients are becoming increasingly keen to demonstrate the value of their experimentation programs beyond any shadow of a doubt.

In such instances, sharing experiment results and running through the usual statistics is not enough. This type of data tells us how each treatment fared against its control, but it does not tell us how these uplifts evolve over time nor how each individual experiment works in conjunction with others.

In a bid to provide data-backed answers to these questions for our clients, we’ve recently run a number of holdouts and even a master experiment (combining all winners into a single experiment and running against the original site) to help our clients demonstrate conclusively that their program is driving significant value for their business. These examples are maybe on the extreme end of the spectrum – long-term holdouts can be quite impractical and master experiments are often fairly labor intensive – but they serve to illustrate a growing desire from stakeholders for more honest accounting of program results.

We believe that throughout 2024 this trend is likely to continue. Budgets are tight and business leaders understandably want to make absolutely certain that their experimentation spend is yielding the results it was assigned to achieve.

3. New legislation to cause a surge in subscription experimentation

As things currently stand, there is a new bill moving through the UK’s House of Lords that is set to have a significant impact on digital markets.

The implications of this bill are quite far reaching, but one particularly significant point (for our clients, at least) relates to subscription brands and the rules surrounding sign-up, renewal, free trials, and more. Specifically, should the bill go through, subscription businesses will be required to:

• provide an easy way of canceling online
• be transparent with costs
• send renewal notices prior to payments being made

While some subscription brands already meet all of these requirements, many do not.

We therefore believe that the bill is likely to stimulate new innovation and experimentation within this space, encouraging subscription brands to develop and optimize new user journeys and policies that keep them on the right side of these new laws.

4. A (partial) return to client-side testing

In 2023, many experimentation organizations made the switch to server-side testing. This came with all kinds of benefits, but it also had its drawbacks.

One particular drawback relates to the increased development time associated with server side testing. Extended development times ultimately mean that testing velocity takes a hit, which in turn reduces an organization’s ability to learn and optimize at speed.

In 2024, we predict that many teams that made a complete switch to server-side testing will begin to shift some percentage of their experiments back to client-side.

In our view, both types of testing have their place in any testing program; the best teams will find a way to get the best out of both.

5. AI will change everything…but maybe not this year

We, like most people, are excited and optimistic (and maybe a little scared!) by the possibilities presented by AI.

In fact, as an agency, we’ve already started putting AI to good use by developing our own machine learning assisted prioritization tool, ConfidenceAI, which is as of a couple of months ago predicting the results of experiments with significantly higher accuracy than the average practitioner.

Screenshot of Linkedin post by our DIrector of Experimentation, Sim Lenz – full post here

While we really do believe that AI has the potential to change the game for our industry, we’re not 100% sure that this is all going to happen in 2024.

On the optimistic side of things, we believe that at some point AI may finally make true, 1-to-1 personalization a reality. After all, if AI becomes sufficiently sophisticated to understand individual user behavior and how best to influence it, why shouldn’t we be able to create completely personalized user experiences that respond to the unique needs and psychologies of each and every user – at scale?

An alternative angle is that AI may well play a key role in facilitating all-team experimentation, allowing product, marketing, design, and data teams to all get the most from experimentation.

Saying this, we’re well aware of the fact that most new technologies fall prey to the Gartner Hype Cycle (see below). While it’s perfectly conceivable that AI may be the exception to this rule, we don’t want to get carried away just yet.

Gartner Hype Cycle

Needless to say, we’ll be keeping a close eye on the AI space in the hope of finding new ways to use this technology to deliver more value for our clients’ programs than ever before.

If last year is anything to go by, the best any of us can do in 2024 is hope for the best, prepare for the worst, and watch in awe as the world’s leading tech companies tear up the rule book once again.

As something of a tech-adjacent space, we believe the experimentation industry has a lot to benefit from incorporating the latest technologies – particularly AI – into the ways we work. The onus is therefore on all of us to keep abreast of these developments and look for opportunities to apply them to our programs to drive more value than ever before.

This would be true even at the best of times, but in an economic downturn, when the need to find a competitive edge is greater than ever, this becomes absolutely imperative.

If you’d like to see how we as an agency are incorporating AI into our processes and approach – plus, how other innovative teams are doing the same – subscribe to our newsletter and we’ll do our best to keep you in the know.