Why do users convert? Introducing our Lever Framework

If we were asked to list the top 5 reasons behind our success as an agency, our emphasis on iteration would surely be one of them. 

Iteration, put simply, is all about taking learnings from early experiments and applying them to later ones.  

In its most basic form, that means that when we have evidence that something works well, we look to do more of it.

But what exactly do we mean by ‘something’ in the phrase ‘something works well’? 

For us, this is what we call a lever, which we define as ‘any feature of the user experience that influences user behavior’. 

If you’re familiar with CRO and experimentation, then none of this will be new to you; but with our Lever Framework, we’ve gone far beyond this basic concept to construct an entire theoretical structure within which different levers can be categorized and ranked. 

This may all sound very abstract, but in practice, our Lever Framework is proving to be an unbelievably powerful tool, letting us:

1. Iterate more effectively within specific experimentation programs
2. Leverage insights more effectively across experimentation programs
3. Collect more meaningful data on experiment win rates

And in addition to all of this, we now also have a comprehensive map of the user experience features that we believe to be effective in eliciting revenue-maximizing behaviors online. 

So, in summary, we’ve come up with an extremely useful framework for understanding and cataloging features of user experience – and throughout the remainder of this blog post, we’re going to share it with you.

Not only that: at the end, we’re also going to show you how we use the framework, sharing the same step-by-step process that recently allowed us to smash the most ambitious revenue target we’ve ever been set.

Contents:

1. The framework structure
   • Why did we go with this structure?
2. The 5 master levers
   1. Cost
   2. Trust
   3. Usability
   4. Comprehension
   5. Motivation
3. The framework in practice: Using the framework to generate millions of $’s worth of additional revenue

 

Quick disclaimer before we start: one of the core premises of online experimentation and CRO is that what works on one website won’t necessarily work on another. That’s why rigorous testing is so fundamental to what we do. 

We’re not arguing against the truth of this claim; all we’re saying is that past experiment results on the same or similar websites can often prove to be a valuable source of insight and inspiration when defining the experimentation roadmap for new websites. Our lever framework will hopefully help you to start doing that more thoughtfully! 

And with that out of the way, let’s get stuck in. 

The framework structure

The framework we’ve developed is a hierarchical tree-like structure that aims to describe changes to user experience at three levels of generality. 

We start with 5 ‘top-level’ levers known as Master Levers. These are:

1. Cost
2. Trust
3. Usability
4. Comprehension
5. Motivation

Next, each of these master levers can be divided into a number of constituent levers, which we simply call Levers. 

And each of these levers can be broken down further into a number of Sub-levers. 

All of this may sound a bit abstract, so here’s a diagram to help make it all a bit clearer.

   As you can see, each Sub-lever has a parent lever, and each lever has a parent master lever.  

So, for example, the sub-lever ‘social proof’ belongs to the lever ‘credibility’ and the master lever ‘trust’.

This hierarchical structure means that any data we collect for the sub-lever is also relevant to its parent lever and its parent master lever as well. 

Returning to the social proof example, then: if social proof has proven to be an effective lever on a client’s website, we can use that information to infer that other sub-levers falling under the ‘credibility’ parent lever might also be worth exploring as well. 

What’s more, we can also infer that other sub-levers and levers that fall under the master lever ‘trust’ may be worth pursuing too. 

Why did we go with this structure?

Before we came up with our lever framework, many of our consultants reported having to make a trade-off between general, broadly-defined levers and specific, narrowly-defined levers:

1. Broadly defined levers, e.g. trust, make it easy to collect huge data samples which we can use to segment our data by other features, e.g. industry.
2. But narrowly defined levers, e.g. social proof, are much more predictive of future results (because they are more similar to one another). 

This hierarchical structure gives us the flexibility to benefit from both broad and narrow levers: we can now use sub-levers for their predictive value while using the more broadly defined levers and master levers to collect larger sample sizes that we use for segmentation. 

Win, win. 

However, the framework isn’t just a tool for better organizing data and prioritizing existing experiment concepts; it’s also a tool for thinking about CRO and drawing a coherent thread from research themes to actionable experiments.

You might ask ‘What kind of problem am I trying to solve? Is it a Usability problem or a Trust problem?’

Once you’ve answered that, you can consult the framework and ask, “What kind of Trust or Usability problem is it?” 

The treelike structure means you can work your way down from a vague understanding of the problem you’re trying to solve to a clear understanding of the kind of experiment you might consider running. 

So, now that we’ve covered the basics, let’s dive into the five main categories of levers.

The 5 master levers

In this section, we’re going to give a whistle-stop tour of the five main categories – or what we call ‘master levers’ – that we use to categorize all of our different levers and sub-levers. 

Unfortunately, we haven’t published a more detailed discussion of each of these individual categories yet – but rest assured, this is something we’re working on right now. If this is something you’re interested in then be sure to follow us on Linkedin – we’ll be releasing this soon!

1. Cost

All levers that come under the master lever ‘cost’ are focussed on how users interpret the cost of a product.

But not all costs are financial. When we say ‘cost’ here we’re referring to how much the user will expect to commit in exchange for the benefits of the product/service they’re considering purchasing. 

So, in other words, all of the levers and sub-levers tied to ‘cost’ are associated with the perceived downsides of the product. These can be:

1. How much money will I have to pay for this product/service?
2. How much time and effort will I have to commit to benefit from this product or service?
3. How much flexibility is there for avoiding or adjusting the anticipated costs?

This master levers structure (seen below) has helped us identify and classify all of these different types of perceived downside within a neat and useful hierarchical structure. 

2. Trust

Our next master lever is ‘trust’, which is about the assessment of risk users make when interacting with a website.

Depending on the severity of this ‘trust’ question, this master lever might relate to several categories of problem:

1. Is this a legitimate website? (or a scam?)
2. Do I believe the website’s claims about the quality of their product/service? (or are they likely exaggerating?)
3. Is there proper protection of sensitive information? (How comfortable do I feel entering confidential information on this site, even if it is a real company?)

3. Usability

In our model, the ‘usability’ master lever is all about how easily users can progress through the website, from arrival on the site through to fulfilling their desired goal, i.e. converting. 

This is very closely aligned with the idea of fluency in psychology: is the user experience arranged such that the thought of not completing their desired action never arises?

This master lever covers several kinds of questions:

1. Does the user know where they are, and where they need to go next?
2. Does the user think taking the desired actions will require a lot of effort? Are they experiencing fatigue?
3. Is the presentation of product options effective in getting users to the right product for them?
4. Is the users’ attention being allocated to the most useful things?
5. Is the users’ on-site progress being properly acknowledged as they complete the actions we want them to complete? 

4. Comprehension

The ‘comprehension’ master lever is all about how well information is delivered and presented on a website. 

It’s about helping users feel comfortable enough in their understanding of the product and transaction that they’re willing to make a purchase.

It therefore considers:

1. Do users understand enough about this industry and general type of product to feel comfortable purchasing this service/product at all?
2. Do users understand everything they need to know about their product of interest on the site to purchase it here?
3. Do users understand everything they need to about the transaction they’re agreeing to in order to convert?

5. Motivation

Motivation is possibly the most varied category of change in this list. 

It’s concerned with what the ‘up-side’ of the product offering actually is: why go through the effort of working through this site? Why pay money, allocate time, and why bother using the product/service at all? 

Fundamentally, it’s asking ‘What’s in it for the user and/or the person they are purchasing it for?’

This can be broken out into several typical questions:

1. Do the descriptions of the product/service make the user feel inspired and excited to have it?
2. Does imagery of the product evoke the right emotions and anticipation to make users excited to have it? 
3. Does the user feel a sense of obligation to convert?
4. Does this product/service give the user access to an imagined (or actual) community?
5. Does it feel particularly important to convert during any given session?
6. Is there an opportunity to experience the qualities of the product before purchasing? And is this opportunity optimal to boost purchase-intent?
7. Does it feel like there’s a wide enough variety of options to choose from, or to which you will gain access? 
8. Is the way that the user will receive access to the product/service compelling?
9. Is the product/service fun or addictive to continue using, and to make further payments in future?

The framework in practice: Using the framework to generate millions of $’s worth of additional revenue

Ok, so, now that we’ve talked through the theory, it’s time to show you how it all works in practice. 

Before we start, it’s worth noting that our lever framework is a tool, and like most tools, it can be applied in a number of different ways to a number of different problems.

We’re still in the process of working out its full range of uses, and the method and examples given below are but one of the many applications that we envision for the framework going forward.

The step-by-step process

The process and examples we’re going to share here are taken from work we’ve recently done with one of our biggest clients – a well-known Saas company.   

To give a bit of context, we’ve been working with this client for a number of years and during that time, we’ve run hundreds of experiments. 

At the start of each year, this company sets us a big target to hit, but this year was different:

This year they set us the most ambitious target we’ve ever been set as an agency, amounting to many millions of dollars of additional revenue throughout the course of the year. 

We’re very confident in our ability to drive results for our clients, but this target marked a new frontier for us – and we knew we were going to have to take the program to the next level if we were going to hit it. 

The framework – with the methodology outlined below – played an instrumental role in our ability to do exactly that. 

In fact, not only were we able to hit the target, but thanks in large part to the lever framework and the methodology to be laid out here, we were able to do this by the end of March, i.e. with 9 months to spare!

Read on to learn about the process that made this possible. 

1. Tag experiments

To begin with, we went through our experiment database and tagged every experiment we’ve ever run for this client based on the sub-lever, lever, and master lever that we used in each one.

So, for example, take the experiment below:

Here, in the variation, we introduced a benefits bar at the top of the page, which included icons and information about some of the core benefits that come with the ‘Premium’ package. 

It’s clear, then, that by emphasizing the benefits of the product, we’re aiming to increase motivation, so this falls within the master lever ‘motivation.’ It’s also clear that we’re trying to emphasize the product’s value statement, so this goes down under the lever ‘value statement’. Finally, it’s clear that this benefit bar emphasizes the benefits of the product, so this goes down under the sub-lever ‘benefit’. 

(For this example, the classification was fairly obvious, but this top-down approach comes in handy when you’re trying to categorize slightly less obvious scenarios).

We tagged the entry for this experiment in our database with the appropriate levers, and then went through the entire back catalog of experiments, doing the same for each. 

By the time you’ve completed this step, every experiment in your database should be tagged with a lever, a sub-lever, and a master lever. 

2. Sort to find high priority levers

Once we’d tagged all of the experiments, we then went about sorting them to see which levers had the strongest success rate. 

Here, we were primarily looking at 2 metrics while conducting this sorting process: 

1. No. of experiments – the more experiments we’d run on a particular lever, the greater the sample size, meaning that we could be more confident in the result as a predictor of future efficacy
2. Average win rate – same as above: the higher the average win rate for a lever, the more confident we could be that it was going to be effective in the future

We then sorted all of our levers into three categories:

1. Exploit – these were levers with a high average win rate that we had already conducted a decent number of experiments on
2. Explore – these were levers that showed promise, but that needed more exploration due to a small sample size (often only one or two experiments)
3. Abandon – these were levers that we’d already done a lot of work with but that had a low win rate

A breakdown of the different levers, as well as the categories we sorted them into, can be seen below.

3. Build experiment roadmap

Once we’d sorted each of our experiments into the three categories above, we then started building out our experiment roadmap for the following quarter.

So, for example, we knew that ‘benefits’ and ‘product depiction’ were particularly effective sub-levers at generating behavior change in this website’s users, so we decided we would create a large number of experiments focussed around these themes.  

With ‘educational content’ and ‘new products’, on the other hand, we’d achieved some early wins focussing on these levers, but having only run a small number of experiments, we were yet to fully validate their effectiveness. We therefore opted to assign 2 and 3 experiments to each of these sub-levers.

4. Couple these results with info about areas of the website with highest win rates

The work we’d done up until this point gave us a very strong foundation on which we were able to start building a data-driven experiment roadmap based on the levers that were historically most effective. 

But we had another trick up our sleeve to make this work even more impactful:

Having conducted hundreds of experiments on this particular website, we’d found that the experiments we ran on certain areas of the site came with higher win rates and greater average uplifts than those conducted on other areas of the site. 

We therefore opted to focus our efforts on those areas that were likely to yield the biggest return. 

With our levers and areas research complete, we now knew which sub-levers and which areas of the website we were going to focus on.  

This still left a lot of work to do – we still had to develop the specific experiment concepts – but at the very least, this preliminary work helped narrow our focus on those experiments that were likely to drive the biggest impact. 

5. Develop concepts based on levers & areas research – and test them!

From here, we started ideating experiment concepts based on the research we’d done so far.

So, for example, in the experiment beneath, we were looking to develop a concept that focussed on the sub-lever ‘product depiction’ and the area ‘Pricing page’.

Our primary KPI was ‘revenue’, and we were looking for a way to increase the proportion of users that purchased the ‘Premium’ subscription relative to the ‘Standard’ subscription. 

But of course, as with any experiment, in addition to our levers and areas research, we also had a number of other research sources that we used to inform the concept:

• Past experiments aimed at increasing the value perception of Premium relative to Standard have been effective at increasing Premium purchases & total revenue
• Also, in a past experiment, by increasing the prominence of Premium plans relative to Standard plans, we’d been able to increase the uptake of the former

Combining all of this evidence with our levers and areas research, we then chose to redesign the Pricing page, giving greater prominence to the Premium package and its features.

And as always, we then ran this newly-designed variation of the page against the original page in an a/b test.

All in all, this experiment increased the number of Premium sales by 40%, which we estimated would amount to a 14% increase in total website revenue if served to 100% of traffic. 

Taken alone, this kind of win is nothing out of the ordinary, but as already mentioned, this result wasn’t a one off. 

In fact, we were able to generate uplifts of this kind with startling consistency, in large part thanks to the framework and process just outlined. 

Final thoughts

As will be evident from the examples presented above, even after you’ve tagged, sorted, and catalogued your experiments using the framework, there’s still a lot of work left to be done. 

But used in conjunction with a robust testing methodology, the framework provides an unbelievably powerful tool with which you can develop high-performing experiment iterations and high-impact roadmaps. 

This blog as it currently exists provides a fairly high-level overview of the framework, but if you’d like to dig into a more detailed exposition of the framework, check out our recent white paper and webinar.