Optimization to Innovation: 
Building a powerhouse experimentation program

The beginnings of our experimentation partnership with Direct Energy largely focused on the plans page.

The plans page was a place where users could shop and select an electricity plan for purchase, comparing and contrasting their choices. Similarly, to shopping for a wireless plan, shoppers could visit the plans page and pick a plan that was suitable to their individual needs.

Experimentation around this area of the website was the large part of our focus from 2017-2019. Testing was primarily focused on optimization, with experiments that iterated off each other and built on their predecessors’ insights.

One of the major benefits of conversion rate optimization is that—done properly—it is a much better alternative to a traditional website redesign. Rather than investing months in a redesign and hoping for a positive outcome, continuous optimization enables you to iteratively redesign a page, funnel, or website, generating uplift and / or learning with every test. That learning piece is one of the most important, and overlooked, parts of CRO.
In one of the first experiments we ran, we wanted to gather information on two key questions:

1. Do customers like more, or less choice? We decided to test this by limiting their available selections from 4 to 3 available plans.
2. Will ordering the plans by price make it easier for users to better comprehend their selections and make a choice?

Variation A insights

This first variant on the plans page uncovered one thing about energy shoppers: having fewer choices provided a worse customer experience. In experimentation, insights are often gained when an experimental variation has a negative impact on performance. This can reaffirm insights and inform strategy before any major changes to the shopping experience are undertaken. By uncovering that Direct Energy customers may prefer choice, future experiments and site designs can be built understanding this concept to avoid making any future changes that can have a negative impact on revenue.

Variation B insights

An addition that proved valuable in this experiment was simply reordering the plans by price. We sought to make it easier for users to comprehend their options and choose a plan.

This saw a 6.42% lift in transactions, an early win in our experimentation partnership. To Conversion, this result indicated an important psychological principle that may be at play with this organization’s customers: processing fluency.Processing fluency is a “metacognitive experience”. That is, “a feeling that accompanies a cognitive process, such as ease of processing a new stimuli or ease of recalling information”. Specifically, processing fluency is “a feeling of ease associated with processing new information, [which] has been shown to affect various judgments”.

In layman’s terms, people who are shopping for energy solutions may be looking for ease, and energy plans aren’t always inherently easy to understand. It’s likely that they know what they want: a more affordable plan, more reliable service, more rebates or “money-back” initiatives, more customization, etc. What matters is how quickly and easily they can find what they’re looking for.

Nobel Prize winner Daniel Kahneman explains this as “System 1 and System 2” thinking in his book “Thinking Fast and Slow”. To paraphrase, users that are operating in System 1 are making fast, automatic, and unconscious decisions. Inversely, those in System 2 are making more effortful, slow, and logical decisions. In short, decision making is easier if we stay in System 1, and processing fluency is a tactic we can use to help keep users in a System 1 state of mind.

Armed with these initial insights and a hunch that showing more plans may be better, we turned our focus to another area of the Plans page – the “All Plans” view.

Back in the first example, we uncovered that this organization’s shoppers MAY prefer choice. Conversion wanted to validate this insight by running a test that eliminated the popular plans tab entirely, showing shoppers all their options at once. The insights from the previous losing experiment helped inform a new strategy for the plan selection, and lead to a new hypothesis.


When you stumble upon a principle that has high elasticity with your audience, you may have unlocked a core customer insight. This type of insight can have a massive impact throughout the entirety of your customer journey, and you’ll likely want to validate it in multiple places.


We developed two experimentation hypotheses:



Hypothesis A: Our first variation had a collection of simple and basic improvements to the UI of the popular plans tab (plans layout themselves) building upon the “clarity” and “processing fluency” insights from the first test. We made the featured plan and promotion clearer and more prominent, and we made the existence of the tabbed structure of the popular and all plans views more apparent as well.

Hypothesis B: We tested an additional variant where we simply got rid of that popular plans tab altogether, and doubled down on user choice, making sure they see all the plans available to them. Shoppers could sort all the plans by the criteria most important to them, such as “Lowest Rate”. The “Most Popular” plan was still featured, including its unique value proposition point of “free power”. However, the large, orange promotional copy was nowhere to be found in this treatment.

This variation increased transactions by over +13%. Interestingly, removing the promotional copy may have increased clarity for shoppers because they no longer had to determine which plan(s) the promotion was associated with. This experiment helped further validate for Conversion how the power of choice would lead to both a better experience for users and more conversions for Direct Energy.

User focused language can make a significant impact in the buyer’s experience and in the case of Direct Energy, help a user select an energy plan that better aligns directly with their needs.


Based on qualitative research and user engagement data, the team zeroed in on five features of this organization’s energy plans that appeared to be most important to shoppers:

• Fixed rate
• Long-term value
• Free electricity promotions
• Bundling
• Renewable energy

We developed an experiment that presented filtering options to receive a recommended plan that would enable users to find a plan that best suits their need while increasing relevance, clarity, and the perception of “personalized” experience, which Conversion hypothesized would result in greater likelihood to convert.

Our hypothesis was confirmed! Visits to checkout increased in conjunction with transactions, indicating a positive result in transactions and traffic through the funnel. We hypothesized that while presenting all the plans available to them – maybe also giving them filter options to narrow down their choices and recommend a plan would increase the ease and clarity of the experience while maintaining the feeling of choice. If successful, we could increase plan selections, checkouts, and transactions. Sure enough, we did! The filters resulted in a significant +6.8% increase in transactions.


The headline for the filters in the winning variation read: “Tell us what’s important to you. Select all that apply, and we’ll recommend the plan that’s right for you.” The icons for each feature were the same, however the labels focused more on the user’s desires.


We were now armed with two other powerful insights:

1. That user-focused language in the winning variant drove significantly more engagement with the filters and the page than non-user focused language.

2. With the engagement data from these new filters we also have a great quantifiable data point of the features users are genuinely interested in! We can see how often users are selecting different filter options and use that data to infer new plan development or user preference.

This lift in conversions demonstrated the strength of a personalized approach towards shopping for a commodity, and how a clarified layout helps bring comfort and clarity to users throughout the checkout experience.

“This test was a fantastic result for us and the team. Not only did we have a great lift in conversion rate which is always what we’re striving for, but we also had a treasure trove of additional insights to glean from user engagement patterns in this test. We learned the importance and hierarchy of these features to users based on their click data. We learned the importance of user-centric language and framing, and we uncovered this new insight that, while users want to see all their options and choices, they also won’t turn down help in narrowing their options down as long as they still feel in control. That’s the new one we’re exploring now: empowering users to make these decisions confidently.”

— James Flory, Director of Experimentation, Conversion

Direct Energy now wanted to expand on more innovative experiments to build new features and functionality to better understand how people really shopped for energy plans – the next phase of the experimentation program helped them answer questions like “Do people think about an energy plan in monthly bill estimates or Kwh? What impact would a bill estimator have on the user experience?” In order to answer these types of questions, we built innovative feature experiments like bill estimators:

Other “next step” questions we looked at addressing were: “What would a more robust plan recommendation wizard look like and would customers use it?” Would it change the way the plans they buy or the features they select?” Building an energy plan chooser gave us new insight into what effect a recommendation based UI would have on customer acquisition.

Other “next step” questions we looked at addressing were: “What would a more robust plan recommendation wizard look like and would customers use it?” Would it change the way the plans they buy or the features they select?” Building an energy plan chooser gave us new insight into what effect a recommendation based UI would have on customer acquisition.

“Your true competition is your customer’s last great experience. Experimentation allows us to rapidly test these bold new ideas before committing time and resources to a complete website or app overhaul.”

— Christopher McCarthy, Manager Digital Paid Media, Direct Energy

From that recommendation wizard test we observed interesting behavior patterns across different devices and user groups that led us to wonder: what would happen if they could see the impact of their selections on their recommended plan live as they were making the selections? We’ve been able to experiment with the grid builder pictured below which has helped us understand how this configuration interface changes a user’s likelihood to purchase, as well as what plans they are more likely to select over others and how that impacts customer retention rates.

Plan builder

These are larger questions that are focused on answering core business and site experience questions and less focused on simple optimizations to the user experience. These innovations have pushed our experiment mix further into the validation and disruption phases and advanced the maturity of our experimentation program together.

These types of experiments have been hugely informative and impactful on building the future direction of Direct Energy’s digital strategy and experience going forward. The impact of these experiments has extended beyond simply improving the digital conversion rate but instead, also helping product teams develop plans, marketing teams fine-tune promotions, and design teams build new experiences from scratch.

Direct Energy’s journey from the inception of a CRO program towards an evolutionary site redesign is one that is filled with critical customer insights that will inform user experience and shopping strategy for years to come.


Our partnership with Direct Energy was able to grow beyond the initial three objectives of their organization. We were able to help them develop a program that aided in testing prioritization, scaled their experimentation practice with our team of CRO experts, and transferred many of our best practices to the team at Direct Energy through knowledge transfer. As we look ahead towards new and innovative experiments that utilize mixed methodologies, Conversion and Direct Energy will build on a cutting-edge program that will help develop initiatives for years to come.