Are you thinking about experimentation too narrowly?

In our recent study of experimentation programs, we interviewed program leaders at 42 different companies to learn how they are defining and applying experimentation. Surprisingly they answered with many distinctly different definitions and applications on how experimentation is best applied in business.

Which begs the question: if leaders have many different definitions of what experimentation means, does that mean some may be applying experimentation/testing too narrowly within their companies? 

To answer, it helps to first consider the growth stages found along a typical life cycle (or S-curve) of a product or business. As a product or business launches, its products sell slowly. As it gains customers and popularity, it may have a time of rapid growth until it eventually slows to relative consistency due to one or more internal or external events.

Put into the context of experimentation, different mindsets typically occur at each of these different growth stages.

For instance, in the early phases, teams take on a more explorative growth mindset as they search for product-market fit. From there, they work on optimizing and fine tuning their product — exhibiting more of an exploitative mindset. Then, as growth starts to peter out, chances are companies need to move into more expansive thinking to continue to innovate and meet customer needs, often by looking for new product opportunities altogether.

We found that a large majority of those we interviewed still primarily limit their thinking around experimentation to the Exploit phase of the S curve — focusing their efforts almost exclusively on conversion rate optimization.

Good experimentation programs are using experimentation to power growth across both the exploration and expansion phases of a product, from validating product market fit, identifying and testing new features, boosting conversion rates, identifying new segments, as well as testing new channels.

Great, leading programs that reported the largest impact apply experimentation across the entire spectrum of growth stages in the life cycles of their products and services–not just the optimization phase. These programs will continually apply experimentation in the expansive phase to innovate and discover entirely new product opportunities.

By doing so, they’re using experimentation to generate multiple S-Curves — continuously finding new opportunities and verticals to be optimized individually to drive growth.

A prime example of this kind of expansive experimentation is at Nationwide. Led by Julia Barham, Nationwide’s Innovation Product Team uses experimentation as a tool to find new, more expansive opportunities for growth.

To deliver on that mandate, Julia’s team tests hundreds of concepts every year to learn about users’ needs, validate (or invalidate) problem-solution fit, and determine what new products and services Nationwide should pursue.

From lightweight concepts to high fidelity prototypes, it’s all about helping to attract and retain customers by putting them at the center of their work.

“We test hundreds of concepts every year to learn about users’ needs and validate (or invalidate) problem-solution fit. We test lightweight concepts to learn if a new feature is attractive and useful to customers, and we also test higher-fidelity concepts when we’re developing a brand new product. All of them are designed to help us attract and retain customers by putting them at the center of our work.”

— Julia Barham – Nationwide

If you’re only thinking about experimentation through the lens of conversion rate optimization, chances are you’re missing out on important opportunities for more expansive growth. It’s not to say conversion rate optimization isn’t important; but if your capacity is limited, it’s important to consider other means by which you can achieve more meaningful growth with the resources you have available, beyond just more iterative product or website improvements.

Learn more on expansive experimentation at Nationwide in our study Maximum Impact, How digital experimentation leaders are doing more with less here.