Optimization to Innovation: 
Building a powerhouse experimentation program

Client Direct Energy
Industry Energy Services

Direct Energy is one of the largest providers of power and gas in the United States. They are a leading energy and energy-related service provider with almost 6 million commercial and residential customers across North America. As a company, Direct Energy has to contend with fierce competition from rival providers in their various geographic locations.


When Direct Energy came to Conversion, they already understood the value of iterative testing to improve competitive differentiation from their visitors. With an inhouse experimentation team that had delivered positive results to date, Direct Energy decided it was time to take their experimentation program to the next level. The three key areas where Direct Energy wanted to improve their optimization program were:

Resource Constraints: Direct Energy wanted to do more experiments but were limited by the availability of internal staff.

Repeatable Framework: Direct Energy wanted more structure to deliver on and show results.

CRO Education: Knowledge transfer from industry experts like Conversion to help elevate their experimentation program.

Throughout this study, we will highlight some of the biggest “A-Ha” moments in our partnership with Direct Energy by taking you through a journey on what a maturing experimentation program goes through over time, and the evolving methodology behind its growth. Our mission, since the outset, has been to understand the unique visitor behaviour by leveraging our proven methodology of statistically valid testing and CRO program optimization.

“Working with Widerfunnel [Conversion] has been a highlight of my optimization career, I first started working with Widerfunnel [Conversion] as an analyst and over the years they have been an amazing resource in not only our company’s growth but my personal growth as I have learned how to run a conversion program and as we at Direct Energy have grown and changed the relationship with Widerfunnel [Conversion] has as well morphing from an educational experience to a truly collaborative team that bounces ideas off of each other and continues to create incredibly fun and thoughtful tests I may have never imagined when I first started.”

— James Hillin , Digital Experience Manager, NRG Energy

Evolution of the plans page through classic CRO: 2017 – 2019

The beginnings of our experimentation partnership with Direct Energy largely focused on the plans page.

The plans page was a place where users could shop and select an electricity plan for purchase, comparing and contrasting their choices. Similarly, to shopping for a wireless plan, shoppers could visit the plans page and pick a plan that was suitable to their individual needs.

Experimentation around this area of the website was the large part of our focus from 2017-2019. Testing was primarily focused on optimization, with experiments that iterated off each other and built on their predecessors’ insights.

One of the major benefits of conversion rate optimization is that—done properly—it is a much better alternative to a traditional website redesign. Rather than investing months in a redesign and hoping for a positive outcome, continuous optimization enables you to iteratively redesign a page, funnel, or website, generating uplift and / or learning with every test. That learning piece is one of the most important, and overlooked, parts of CRO.
In one of the first experiments we ran, we wanted to gather information on two key questions:

  1. Do customers like more, or less choice? We decided to test this by limiting their available selections from 4 to 3 available plans.
  2. Will ordering the plans by price make it easier for users to better comprehend their selections and make a choice?

Variation A insights

This first variant on the plans page uncovered one thing about energy shoppers: having fewer choices provided a worse customer experience. In experimentation, insights are often gained when an experimental variation has a negative impact on performance. This can reaffirm insights and inform strategy before any major changes to the shopping experience are undertaken. By uncovering that Direct Energy customers may prefer choice, future experiments and site designs can be built understanding this concept to avoid making any future changes that can have a negative impact on revenue.

Variation B insights

An addition that proved valuable in this experiment was simply reordering the plans by price. We sought to make it easier for users to comprehend their options and choose a plan.

This saw a 6.42% lift in transactions, an early win in our experimentation partnership. To Conversion, this result indicated an important psychological principle that may be at play with this organization’s customers: processing fluency.Processing fluency is a “metacognitive experience”. That is, “a feeling that accompanies a cognitive process, such as ease of processing a new stimuli or ease of recalling information”. Specifically, processing fluency is “a feeling of ease associated with processing new information, [which] has been shown to affect various judgments”.

In layman’s terms, people who are shopping for energy solutions may be looking for ease, and energy plans aren’t always inherently easy to understand. It’s likely that they know what they want: a more affordable plan, more reliable service, more rebates or “money-back” initiatives, more customization, etc. What matters is how quickly and easily they can find what they’re looking for.

Nobel Prize winner Daniel Kahneman explains this as “System 1 and System 2” thinking in his book “Thinking Fast and Slow”. To paraphrase, users that are operating in System 1 are making fast, automatic, and unconscious decisions. Inversely, those in System 2 are making more effortful, slow, and logical decisions. In short, decision making is easier if we stay in System 1, and processing fluency is a tactic we can use to help keep users in a System 1 state of mind.

Armed with these initial insights and a hunch that showing more plans may be better, we turned our focus to another area of the Plans page – the “All Plans” view.

UI clarity and simplicity matter – all plans clarity redesign

When designing a user experience, it is incredibly important to keep in mind that users will be accessing your site on mobile, laptops and tablets as well. A UI that avoids clutter, is simplistic and informative to the shopper can strengthen your user experience and help boost conversion.

For this experiment, the control UI appeared like the below. The CTAs were placed below the KWH rate, rather than in line with the plan row creating a lot of wasted empty space. This “wasted space” created a very cluttered and condensed plan description section to the right. Conversion hypothesized that this potentially unclear layout was confusing to the visitors and could be having a negative effect on conversion rates.

Our hypothesis was proven correct; a “Clarity Redesign” of the plans page “all plans” view saw a substantial increase in conversions by creating a more simplistic experience that reduced clutter by changing the placement of the CTA’s on the page. The overall UI improvements highlighted the power behind creating an experience that focused on a higher processing fluency with a more simplistic and 
user-friendly UI.

Removing the “popular plans” – expanding choice

Back in the first example, we uncovered that this organization’s shoppers MAY prefer choice. Conversion wanted to validate this insight by running a test that eliminated the popular plans tab entirely, showing shoppers all their options at once. The insights from the previous losing experiment helped inform a new strategy for the plan selection, and lead to a new hypothesis.

When you stumble upon a principle that has high elasticity with your audience, you may have unlocked a core customer insight. This type of insight can have a massive impact throughout the entirety of your customer journey, and you’ll likely want to validate it in multiple places.

We developed two experimentation hypotheses:

Hypothesis A: Our first variation had a collection of simple and basic improvements to the UI of the popular plans tab (plans layout themselves) building upon the “clarity” and “processing fluency” insights from the first test. We made the featured plan and promotion clearer and more prominent, and we made the existence of the tabbed structure of the popular and all plans views more apparent as well.

Hypothesis B: We tested an additional variant where we simply got rid of that popular plans tab altogether, and doubled down on user choice, making sure they see all the plans available to them. Shoppers could sort all the plans by the criteria most important to them, such as “Lowest Rate”. The “Most Popular” plan was still featured, including its unique value proposition point of “free power”. However, the large, orange promotional copy was nowhere to be found in this treatment.

This variation increased transactions by over +13%. Interestingly, removing the promotional copy may have increased clarity for shoppers because they no longer had to determine which plan(s) the promotion was associated with. This experiment helped further validate for Conversion how the power of choice would lead to both a better experience for users and more conversions for Direct Energy.

Copy and conversions: reducing anxiety for maximum impact

The power of copy is sometimes glossed over and not recognized for its power to enhance a brand or website experience. The right words can inform, inspire action, and even reduce anxiety for many users. A call to action, if ambiguous or overtly imposing can create anxiety and cause users to shy away from the next step in their shopping journey.

Working with Direct Energy, it was Conversion’s hypothesis that some users were hesitant to click the CTA on the plans page due to the committal language. The CTA, which read “Order Now” in control, felt very committal – like clicking the CTA would initiate some sort of transaction when, all it did was select the plan and move a user to the first stage of a checkout experience where they would need to enter their address and other conventional signup information. However, this test sat in the backlog for two years, because it wasn’t seen as “exciting” as other ideas being put forth, so it kept getting bumped – but its “Ease” was ranked so highly, that it should have been run sooner – it’s so clearly impactful in the end, and hit on some key elasticity.

The control CTA looked like the below:

A bold, forthright statement with copy that Conversion decided would be worth testing against. We tested several copy variations that softened the tone and were more in tune with what the next step of the checkout journey would look like:

The results clearly and profoundly confirmed our hypothesis; users wanted a less committal, more inviting CTA that helped see them to the next step of their journey. Variation C’s “Select Plan” copy was a clear winner, capitalizing on reduced anxiety and thus friction at the CTA point, ultimately increasing the volume through the funnel to the checkout and increasing transactions. As resounding success evidenced by the above. A CTA copy test can still be hugely impactful if it’s a key part of the customer journey and if its copy is overlooking some key LIFT points (relevance and anxiety in this case).

In experimentation, there sometimes exists the notion that CTA copy tests are “simple” tests that often get little attention due to their perceived simplicity. While CTA copy tests aren’t exactly innovative or game-changing and they don’t often “wow” internal stakeholders, with an estimated 24% increase in transactions, this very simple change was virtually a no-brainer to implement.

Filters and focused language: a difference maker

User focused language can make a significant impact in the buyer’s experience and in the case of Direct Energy, help a user select an energy plan that better aligns directly with their needs.

Based on qualitative research and user engagement data, the team zeroed in on five features of this organization’s energy plans that appeared to be most important to shoppers:

  • Fixed rate
  • Long-term value
  • Free electricity promotions
  • Bundling
  • Renewable energy

We developed an experiment that presented filtering options to receive a recommended plan that would enable users to find a plan that best suits their need while increasing relevance, clarity, and the perception of “personalized” experience, which Conversion hypothesized would result in greater likelihood to convert.

Our hypothesis was confirmed! Visits to checkout increased in conjunction with transactions, indicating a positive result in transactions and traffic through the funnel. We hypothesized that while presenting all the plans available to them – maybe also giving them filter options to narrow down their choices and recommend a plan would increase the ease and clarity of the experience while maintaining the feeling of choice. If successful, we could increase plan selections, checkouts, and transactions. Sure enough, we did! The filters resulted in a significant +6.8% increase in transactions.

The headline for the filters in the winning variation read: “Tell us what’s important to you. Select all that apply, and we’ll recommend the plan that’s right for you.” The icons for each feature were the same, however the labels focused more on the user’s desires.

We were now armed with two other powerful insights:

  1. That user-focused language in the winning variant drove significantly more engagement with the filters and the page than non-user focused language.

  2. With the engagement data from these new filters we also have a great quantifiable data point of the features users are genuinely interested in! We can see how often users are selecting different filter options and use that data to infer new plan development or user preference.

This lift in conversions demonstrated the strength of a personalized approach towards shopping for a commodity, and how a clarified layout helps bring comfort and clarity to users throughout the checkout experience.

“This test was a fantastic result for us and the team. Not only did we have a great lift in conversion rate which is always what we’re striving for, but we also had a treasure trove of additional insights to glean from user engagement patterns in this test. We learned the importance and hierarchy of these features to users based on their click data. We learned the importance of user-centric language and framing, and we uncovered this new insight that, while users want to see all their options and choices, they also won’t turn down help in narrowing their options down as long as they still feel in control. That’s the new one we’re exploring now: empowering users to make these decisions confidently.”

— James Flory, Director of Experimentation, Conversion

Part 2: The site redesign

Why redesign? Understanding Direct Energy’s decision

After 3 years of optimization and continued improvement and lift through experimentation, Direct Energy had reached a point in 2019 where they had decided that their brand was due for a bit of a refresh. Brand refreshes are sometimes needed to take large leaps forward in customer acquisition and overall brand competitiveness in the marketplace. Part of this refresh meant that a website redesign was on the horizon. Our partnership and years of incrementalism and learning through experimentation leading up to this point had armed the team responsible for the redesign with a substantial amount of insight about the user and their preferences of the experience to make really informed design decisions and start from a place of “data-drive redesign”. An effective CRO program, like the one developed at Direct Energy, can help inform the next large, bold, iteration of experimentation: the website redesign.

“Experimentation allows us to grow rapidly and in the right direction. Gone are the days of complete web overhauls that take multiple years and to hope for the best upon launch. Each page, each button, each menu has been tested so that the next evolution of a website is a culmination of many new experiences, all proven.”

— Christopher McCarthy, Manager Digital Paid Media, Direct Energy

Our work to date gave the redesign team a better understanding of:

  • The power of choice
  • Site Filters – when to include and exclude them
  • Copy and Terminology Usage
  • Balancing processing fluency with new ideas

The customer insights we were able to discover together through the inception of their CRO program helped inform this redesign with learnings from all the experiments conducted to this point. The redesign is, in itself, a large experiment. Direct Energy acknowledged the risk of an experiment of this size and have committed to measuring its impact on the business.

With the support of Conversion, the team ensured the redesigned elements made use of key experiment findings and tweaked and tested them as they developed the redesigned pages. And through this thoughtful approach, were able to launch a redesigned digital experience with no negative impact to their key customer journey metrics – a difficult feat in the world of redesigns.

“I think we have become a more data driven and customer centric company. Turning to customers for research and testing to verify has been great. I think beyond the obvious benefits of increased sale and customer engagement, testing has also created an almost adventurous mindset within the team to try new ideas and not be afraid to be wrong.”

— James Hillin, Digital Experience Manager, NRG Energy

A mature experimentation program containing a foundation of knowledge built on previous experiments allows a company like Direct Energy to lean into a more disruptive, and innovative testing strategy once the redesign is deployed. The maturation and progression of Direct Energy’s experimentation program follows the steps listed below:

After the redesign, the experimentation program with Direct Energy became less about the small tweaks and “optimizations” and more about user experience and “how to shop for energy”. The experimentation program is now ready to focus now on more innovative and advanced experimentation, taking novel approaches to experimentation to help the brand change the way people shop for energy.

Mature experimentation: The user experience at Direct Energy

Direct Energy now wanted to expand on more innovative experiments to build new features and functionality to better understand how people really shopped for energy plans – the next phase of the experimentation program helped them answer questions like “Do people think about an energy plan in monthly bill estimates or Kwh? What impact would a bill estimator have on the user experience?” In order to answer these types of questions, we built innovative feature experiments like bill estimators:

Bill estimator

Other “next step” questions we looked at addressing were: “What would a more robust plan recommendation wizard look like and would customers use it?” Would it change the way the plans they buy or the features they select?” Building an energy plan chooser gave us new insight into what effect a recommendation based UI would have on customer acquisition.

Other “next step” questions we looked at addressing were: “What would a more robust plan recommendation wizard look like and would customers use it?” Would it change the way the plans they buy or the features they select?” Building an energy plan chooser gave us new insight into what effect a recommendation based UI would have on customer acquisition.

Chooser – user experience

“Your true competition is your customer’s last great experience. Experimentation allows us to rapidly test these bold new ideas before committing time and resources to a complete website or app overhaul.”

— Christopher McCarthy, Manager Digital Paid Media, Direct Energy

From that recommendation wizard test we observed interesting behavior patterns across different devices and user groups that led us to wonder: what would happen if they could see the impact of their selections on their recommended plan live as they were making the selections? We’ve been able to experiment with the grid builder pictured below which has helped us understand how this configuration interface changes a user’s likelihood to purchase, as well as what plans they are more likely to select over others and how that impacts customer retention rates.

Plan builder

These are larger questions that are focused on answering core business and site experience questions and less focused on simple optimizations to the user experience. These innovations have pushed our experiment mix further into the validation and disruption phases and advanced the maturity of our experimentation program together.

These types of experiments have been hugely informative and impactful on building the future direction of Direct Energy’s digital strategy and experience going forward. The impact of these experiments has extended beyond simply improving the digital conversion rate but instead, also helping product teams develop plans, marketing teams fine-tune promotions, and design teams build new experiences from scratch.

Part 3: A culture of experimentation and decision making at Direct Energy

Experimentation at Direct Energy has grown from simply having an outside partner agency run tests to improve the conversion rates of their digital properties, to expanding the understanding of experimentation as a methodology of making more informed decisions across the organization.

“A culture of testing is centered around innovation and that’s what exactly Widerfunnel [Conversion] has helped Direct Energy with. Throughout the years we had tremendous success with measuring our conversion rate alongside our overall financial impacts but I think the biggest impact was the organizational mindset shift where we want to test everything before bringing it in front of our customers.”

— Ethan Rahman, Sr. Manager UX/UI & Digital Development, Direct Energy

Today, different teams make use of experimentation and our partnership to ask key business questions of the experimentation program. Marketing and content teams are using experimentation insights to create better content, Product teams are using experimentation to develop better products, and the digital teams continue to use it to enhance digital experiences.

“Experimentation forces us to measure our results and the success of our proposals. WF brought us the culture of experiments and pushed our boundaries and helped us make better decisions for our customers and business.”

— Ethan Rahman, Sr. Manager UX/UI & Digital Development, Direct Energy

“Collaboration, having representatives from different teams in the room always added a special feel to the experience of creating a test and everyone’s willingness to take a risk or be wrong made for some exciting collaboration.”

— James Hillin, Digital Experience Manager, NRG Energy


Direct Energy’s journey from the inception of a CRO program towards an evolutionary site redesign is one that is filled with critical customer insights that will inform user experience and shopping strategy for years to come.

Our partnership with Direct Energy was able to grow beyond the initial three objectives of their organization. We were able to help them develop a program that aided in testing prioritization, scaled their experimentation practice with our team of CRO experts, and transferred many of our best practices to the team at Direct Energy through knowledge transfer. As we look ahead towards new and innovative experiments that utilize mixed methodologies, Conversion and Direct Energy will build on a cutting-edge program that will help develop initiatives for years to come.

Want to work with us?

We work with everyone from small startups to some of the biggest brands in the world.

Get in touch and see how we can help you.

Contact us