From a one off project to an ongoing, multi-team experimentation program

Client The Times

Industry Media

The Times initially brought us on to deliver 40+ experiment concepts aimed at driving results within the acquisition area of the business. However, as the early successes began to roll in, we were ultimately hired to run two separate, full-scale experimentation programs across the entire business – achieving some fantastic results in the process.

Opportunity

The Times operates in a highly competitive and fragmented digital news media landscape. Its role as a provider of reliable and extensive non-partisan news is squeezed by several external pressures, with social media and other emerging forms of media, e.g. streaming services, taking up an ever growing slice of the ‘attention economy’.

In order to stay ahead of the competition, the quality content of the Times has to be supplemented with a high quality of user experience and a savvy digital strategy.

The Times therefore launched a challenge to several CRO agencies: to build a backlog of 40+ high impact experiments in less than 8 weeks, which could drive their experimentation roadmap for the acquisition of new customers.

Ultimately, after a rigorous selection process, our expertise and drive convinced key decision-makers at The Times that we could deliver the strongest return on investment.

Delivering beyond a backlog of experiments

This task required a different approach from our usual process at Conversion, which involves drawing our strongest insights from experiment data and doubling down on early wins to drive higher impact experiment roadmaps. Nonetheless, we approached this challenge with enthusiasm, and quickly became fascinated with the opportunity to understand and advance CRO in the news media.

We started off with an in-depth research and insights plan, which combined multiple data sources: historical experiment results, interviews, usability findings, analytics reviews, heat maps, and external research. This gave us our data-driven start point for understanding and prioritizing the most important motivations, barriers, areas, and audiences of The Times.

We collated this data through multiple ideation sessions, generating bespoke hypotheses and themes of experimentation most relevant to the Times’ audience and value proposition. These robust insights drove a bounty of experiment ideas. By working closely with our in-house experts on UX design to generate concepts and iteratively refined implementations, we were quickly able to populate the requested roadmap with high-impact, data-driven experiments.

Just under 6 weeks after starting the project, we delivered the first batch of 22 experiments, followed one week later by a second session where the other 21 experiments were presented. The teams attending were highly engaged and provided valuable context and additional information which helped us select and prioritize the experiment concepts which would be ready for swift execution.

In parallel to our work delivering the experiments, we had several presentations and training sessions with The Times’ wider team in which we covered a broad curriculum on how to run strategic CRO experimentation programs. We built on their existing experience with experimentation by teaching them the most scalable and reliable approaches that we’ve cultivated during our 10+ years running advanced CRO programs.

A wrap up session was organised in week 8 in which we presented finalised concepts, alongside a ‘how to’ playbook for effective experimentation. From here, The Times began running our recommended experiments, and the ROI potential quickly became clear following a series of early successes.

Running a cross-team experimentation program

A combination of limited resources dedicated to experimentation at The Times acquisition and the strength of our early collaboration led to a ‘one-off’ project being extended to a complete CRO program.

In the first 6 months of the program, we increased our trial sign-up KPI by an impressive 28%. We saw substantial impacts from simple experiments focused on removing friction from user journeys, increasing relevance, and improving the presentation of The Times’ offering. We also introduced a basic form of personalization for returning users, which turned out to be our biggest winner, opening up potential for wider personalization in the future.

The acquisition team at The Times have done an excellent job of voicing the program’s success and championing experimentation within the business. The growing interest and trust in experimentation has allowed us to run bolder, business-critical experiments to inform key strategic decisions about product and pricing.

Ultimately, this excellent win-rate, wealth of accumulated understanding, and maturation of their culture of experimentation has since led to The Times extending the Conversion acquisition program for another year.

Moreover, this success yielded the opportunity to enter into a new collaboration with the ‘engagement’ team at The Times. This team has a completely different set of KPIs, focussed on increasing content consumption and active days on The Times website among existing customers. This has given us the opportunity to cultivate an ‘end-to-end’ understanding of their customers, from converting prospective customers to free trials through the customer lifecycle of engagement and renewal.

“The team at Conversion really understand our business needs and challenges. Their client service is exceptional, but most importantly they deliver results.”

Victoria Mead Head of Subscriptions

Increasing win rate with Neurolab behavioral insights

In the Autumn of 2021, we received yet another opportunity to extend our understanding of prospective and existing customers at The Times. This came in the form of our dedicated behavioral science product, NeuroLabs, which looks to go beyond conventional research methods by collating user interview data with facial expression analysis and scientifically backed ‘persona’ construction. This has enriched our understanding even further, producing a wave of novel experiment concepts and directions to enhance the promise of the program for the upcoming year.

All in all, we feel privileged to have had the opportunity to work within such a fascinating and important industry, and alongside such a diligent and open-minded team of collaborators, who have helped make our work at The Times such a success.

At this time, we’re continuing in our successful collaboration with The Times, and look forward to another year of increasingly bold, impactful experimentation.