What marketing executives need to know about A/B Testing and CRO
Marketing professionals commonly think that Conversion Rate Optimization or A/B testing tools will adequately address all of your problems, but that’s rarely the case. To succeed, it’s imperative that you fully understand CRO and A/B testing basics, build on your knowledge continuously and work toward leveraging a more fulsome approach overtime.
Once you have the basics of CRO and A/B testing explained, and you will clearly understand the advantages of growing a mature and strategic practice with strategy and tools, you will be able to identify where additional support is required to drive deeper buildable insights.
CRO tools vs. strategy
By simply putting your trust into a tool, you may potentially end up with muddy data, inconclusive test results, and limited insights. Tools and platforms are helpful to manage and keep track of your insights and data; however, they will not successively teach you how to identify trends, patterns or lay the foundation for best practices.
How testing and optimization impact your bottom line
Taking a strategic approach to CRO or A/B testing can dramatically improve the customer experience and increase revenue. However, keep in mind, simply changing button colours or performing random tests may bring initial insights or perceived wins for a small period of time, but this isn’t a long-term scalable strategy.
You must consider what you are testing, how you are testing, and why you are testing to design successful experiments and connect you to customer’s motivations or problems—answering “the why” may help you identify the root of the business problem. Generally, we would recommend taking a Mixed Methods approach to uncover the most insight for your business problem.
Good to know:
An unstructured testing program will yield inconclusive tests and ultimately create a stunted experimentation program that will not instill confidence with your organization. Conversion optimization using A/B testing has become standard practice at virtually every high-traffic online business today. But, not all organizations are seeing significant impact from their experimentation programs.
At Conversion, we achieve a 40% win rate on all tests, compared to an industry average of 15%-25% (Source: VWO)
A/B testing and experimentation
What is A/B testing?
There are varying complexities to A/B testing and CRO programs. In a nutshell, it’s a method of measuring the performance of different versions of a website or digital experience to determine which is best.
Why should you A/B test and create an experimentation program? (The Benefits of A/B Testing)
To produce desirable results that create more wins and positively impact the budget, structured systems should be implemented.
While the purpose of A/B testing is to gain a better understanding of what converts now, you will be able to derive customer insights from testing to build an improved customer experience for products and services you roll out in the future. Ideally, as your practice matures, you should consider complementary methodologies in your A/B testing program to help you gain broader insights and drive your experimentation program.
Marketers that employ a rigorous experimentation practice experience benefits like reduced bounce rate, increased conversion rate, and higher conversion values. On a more strategic level, you can build a world-class optimization program that will:
- Create more relevant experiences for your customers
- Drive consistent optimization results
- Lift revenue and customer insights
What can you A/B test?
If you’re looking for A/B testing ideas, we suggest you consider testing the homepage, service pages, product landing pages (PLP) and product display pages (PDP) and checkout page of your website or application—think low-hanging fruit.
To determine which pages or experiments to prioritize, refer to the PIE framework developed by Chris Goward.
Example of a Conversion PIE prioritization analysis
At Conversion, we use fractional factorial design to ensure that we maximize rewards for running A/B tests.
Good to know:
Recently, our test data suggests that smaller isolated tests (incremental changes) may be just as impactful as running larger-scale tests. However, the mix of the types and sizes of experiments should be purposeful and align with your organization’s goals.
In general, A/B testing and experimentation provides us with answers to key business questions or solutions to customer problems. It’s important to think differently and approach problem-solving in new ways.
When designing your next A/B test, think about including the following elements in your test:
To run an A/B copy experiment on your website, you can make variations in formatting, positioning, tone, and length to test how your customers respond. Various formats can be tested; for example, you can change the tone of the call to action on your CTAs.
An example of A/B testing ad copy for Precision Nutrition
Design and layout
A/B testing is also used in the UX and design practice. By creating variations with layout and design, you can gauge what experience your customers respond to best. Test results can also inform information hierarchy, which can be critical to market your service or product. For example, it can determine the story that you are telling to customers.
A/B testing can help determine factors like the best overall navigation experience for your visitors and the most favourable wayfinding paths, i.e. the typical courses users take to find information to help them along their customer journey.
Experiments can reveal a lack of clarity around the primary call-to-action in the navigation, if there was a general lack of urgency, if the menu drop-down was ambiguous or confusing, or if a visitor is ready to spend money, but it’s not clear how to purchase. Testing navigation variants will help you determine the most effective way to design a revenue-driving funnel from all parts of the site.
Lead form optimization is an art. You will want to examine several aspects to plan your tests accordingly.
You will want to consider testing:
- Form length
- Designing an experience without the use of pop-ups
- Single-step vs. multi-step form
- Progress bars or indicators
Be keen to notice user behaviour patterns and design an experience that presents minimal barriers to conversion.
CTAs (Call To Action)
Knowing which CTAs resonate with your customers is a difficult thing to determine—this is where testing comes in handy. Remember, CTAs are much more than just a simple phrase we append onto an offer that we hope will serve as a powerful motivator. Your CTA A/B testing should be clear, concise, and your customers’ should easily understand your unique value proposition. Think about copy variations, placements you can test, and your on-page presentation. Lastly, remember the CTA should also be clear on where it’s going to take users. Too often, we see businesses with CTAs that lack clarity.
To move visitors into a sale naturally, you must design a conversion funnel. An optimized conversion funnel can improve your user experience and provide valuable insights about customer behavior. Additionally, conversion funnels will help create a healthy pipeline, contribute to a higher conversion rate, and positively impact your company revenue.
How to design A/B tests
Marketers often use the term ‘A/B testing’ to refer to marketing experimentation in general. But there are multiple ways to structure your experiments. A/B testing is just one of them.
With various frameworks to choose from, selecting how to design your A/B testing can range from very simple to highly complex. Some of the experimentation structures include A/B testing, A/B/n testing, full factorial or multivariate (MVT), and fractional factorial design.
Step 1: Research
Before you enter into any testing program, it is essential to assess the performance of your website or eCommerce platform. Collecting quantitative and qualitative research will help you get a high-level view of your customers’ experience, allow you to design your testing program, and ultimately prepare you to make informed business decisions. Consider analyzing your Google Analytics data, employ a heatmap tool, and conduct customer surveys to start. Allocate an appropriate amount of time to assess and interpret the data.
At Conversion, we’ve developed a proprietary framework called the to guide our research. This framework consists of an Explore phase and a Validate phase, with a central area where they meet—this is where you will uncover valuable growth opportunities and insights. We leverage user research, digital analytics, and secondary research from Behavioural Science to inform the next steps in the Explore phase.
Step 2: Observe and formulate the hypothesis
A well-structured hypothesis provides insights whether it is proved, disproved, or results are inconclusive. You should never phrase a marketing hypothesis as a question. It should be written as a statement that can be rejected or confirmed.
“Coming up with something you want to test isn’t hard to do. Coming up with something you should test can be hard to do.”
— Chris Goward
Further, it should be a statement geared toward revealing insights – with this in mind, it helps to imagine each statement followed by a reason.
Step 3: Create variations
Step 4: Run test
A/B testing principles aren’t as straightforward as they might look. Prior to running your test, you must determine the method you will be employing.
A/B Testing considerations
If you want to change an element on multiple pages, you could test it across the board with:
- Single page
- Multi-page testing or funnel testing
- Template testing (using the same template but different content)
Conversion Rate Optimization (CRO)
What is Conversion Rate Optimization?
Conversion rate optimization (CRO) is the process used to increase the percentage of visitors’ who take a desired action on your website or app, such as a transaction, enrollment, or subscription to a newsletter.
Example of an ecommerce website
Research and development help you reach the desired result through repeated cycles of testing.
Concerned with high impact areas
By prioritizing high-impact, high traffic areas of your site first, you can maximize potential ROI from AB testing and increase the speed of which your tests run.
Tailored to the goals of the consumer
CRO can help to identify and solve customer problems within the digital UX. By aligning with the goals of the customer, CRO can help improve key aspects of the customer experience.
Challenges of A/B testing and CRO
What are the challenges of A/B testing and CRO?
No matter what experience level you have, there will be A/B testing challenges to navigate.
Deciding what to test
With so many aspects of the website to optimize, how do you know which area to start at first? To help you determine which pages or experiments to prioritize, refer to the PIE framework that Chris Goward developed The PIE Framework.
The total number of users on your site, your sample size, will determine how quickly your company can test or if your company is in a position to test at this time.
If your results are disappointing, it may not only be what you are testing – it could be how you are testing. While there are several factors for success, one of the most important to consider is test design or Design of Experiments (DOE). DOE is rooted within statistical theory and defines how experiments are planned to maximize learnings and performance.
There are several differences between split testing vs. A/B testing. However, the most critical dissimilarity is that split testing requires you to run two live versions of the site, both hosted on different URLs.
Multivariate (MVT) or Full Factorial Test Design
With multivariate A/B testing, you change several individual sections of the website isolated one against another, by mixing and matching every possible combination.
Fractional factorial test design
Rather than creating a variation for every combination of changed elements (as you would with MVT), you can design your experiment to focus on specific isolations that you hypothesize will have the biggest impact. By building each subsequent variation off of the previous one, you’re able to isolate each change and measure the total impact of all the changes combined maximizing learnings.
Analyzing test results
Even with the best A/B testing setup for optimization, if you aren’t tracking the right metrics to analyze the data, your hard work will go to waste.
Learning from losing experiments
Giving up after seeing losing AB tests is a common reason as to why CRO initiatives fail. If you are consistent, learning and iterating on previous AB tests, you can continue testing until you find the right optimizations.
Metrics & KPIs
What data should you collect?
The A/B testing metrics you will want to report on will depend on your business objectives and your business model. Consider that SaaS, eCommerce and lead gen activities have different KPIs.
Business goals and KPIs
Key performance indicators must be a measurable value that demonstrates how well the company met business objectives. It is essential to the success of your CRO program to outline your KPIs and desired outcome in advance. Consider if your goals are short- and long-term.
Typically, goal completions will be key indicators of how your experiments perform. These metrics will help evaluate the impact of your tests and determine whether the new experiences should be implemented on your website.
If you are running an experiment, revenue is arguably one of the key metrics you should be tracking. Even if you increase the order quantity/goal completions, the experiment that you have shouldn’t negatively impact revenue.
Cost Per Conversion
This KPI will be relevant for you if your program has a paid traffic component. CPC is the cost of acquiring one customer—the lower the amount, the more profitability you can expect. If you don’t have a proportionate amount of paid traffic, you can track cost per conversion, but attributing a lower CPC to your experiment may not be straightforward.
Average Order Value (AOV)
To determine this metric’s value, you will want to divide the total revenue by the number of orders. Depending on your business model, you can also track additional metrics, for example, ARPU, MRR and LTV.
Shopping cart abandonment
This is one of the metrics that no one wants to see increase. To improve ROI, you must boost the number of customers that follow through on the sale. At Conversion, we track the entire journey vs just focusing on cart abandonment. In addition, our strategists include the metric quantity per order since that helps to improve revenue.
A/B Testing Tools
What are A/B testing tools?
You can find platforms that can help you run A/B testing, multivariate testing and include website personalization capabilities. You will also notice that there are platforms dedicated to improving your collaboration with your team, grow your experimentation program efficiently, and share insights company-wide. Choosing the right tools for your team can be daunting since there are quite a few on the market.
No matter the size or maturity of your experimentation practice, we recommend that you start with a collaboration tool that will help you prioritize hypotheses, store experiment details, comment on tests, and share results and insights with the broader organization.
Are you using the right tools?
The best A/B testing tools are going to help you run tests, make hypotheses and analyze the data.
Choosing the best A/B testing tools for you
As you analyze A/B testing website tools, consider these factors.
- What is the skill level of my team?
- What technical resources will be required?
- What skills will be required to use the software?
- What level of support is Available?
- What volume is needed to run a test?
- Will the software slow down my site?
- How much will the tool cost?
- What are my business objectives?
- What other tools will be needed?
As you progress in the journey, you will need to grow your tech stack.
Tools and business objectives
Deciding what tools and A/B testing software you need will depend on your goals. There are many website A/B testing tools needed to get a full picture; it’s more complex than just running a single tool.
The best A/B testing platform might integrate these objectives.
- User tests
- Business intelligence
- Statistical significance calculator
- Web analytics
- User analytics
- A/B testing
Thankfully, there are many big names producing some of the best A/B testing software.
Heatmaps visually represent where users click, move, and scroll on your site. Source: hotjar
Some examples of A/B testing software
Organization challenges to address
Politics and people
You need your team to be on-board with the plan and there must be trust with management.
Insufficient budget allocation
You can’t afford to neglect CRO, even if it looks like a higher cost than what you were expecting. CRO budgeting is one line item that must be allocated for if you want to succeed.
Creating a testing culture
By building a culture of experimentation, you encourage your team to think outside the box and embrace failing tests as a learning opportunity.
Growing & maintaining a testing culture
Consider democratizing your experimentation program. Enable teams that aren’t directly involved in experimentation to participate and run their own tests/hypotheses via the program.
It is always favorable to grow experimentation in your organization and foster inclusivity.
Resources & structure to support CRO
If you don’t have the appropriate resources to fill essential roles on your team, partner with a trusted CRO agency or resource to ensure you’re getting the proper support and meeting best practices.
Building your own experimentation team
Why you need a team
If you are invested in scaling and maturing your CRO practice, it’s going to take a dedicated team to get you there.
How to scale your A/B testing team
To have a successful experimentation practice, you need to create a testing team of qualified professionals, which may increase your overhead costs.
A well-constructed team consists of the following roles:
- Marketing Strategist
- Data Analyst
- QA Tech
Anything less than this could lead to disaster and wasted money.
Hiring a CRO Agency
Do you need to hire a CRO agency?
If your company doesn’t have the in-house resources to handle your experimentation programming, it might be time to explore working with a conversion rate optimization agency. The best fit CRO agency is going to remove the burden from your shoulders and get results that improve profitability.
How to pick the right experimentation partner for your business
User-driven, experience-driven or data-driven?
What does the A/B testing agency use to create a strategy?
You want a conversion rate agency that offers all the services you need in a package, so you won’t have to hire multiple providers.
Ask the CRO marketing agency to show you case studies of other jobs that have recently been completed.
As an eCommerce CRO agency, the steps and strategy should be plainly laid out so you can understand them.
Research & discovery
Ensure the agency has the experience to handle the research aspect of your testing.
Align with your development partners
Your CRO agency should have experience working hand-in-hand with the other trusted partners in your network.
Focus on long-term or short-term results
Ideally, you want a company that’s committed to providing short-term results for a rapid ROI, but are also committed to maturing your experimentation practice.
How Conversion can help
What you can gain from an experimentation platform
With an experimentation platform, you gain scientific results that drive your success. There’s no room for guessing when it comes to your business and goals.
Conversion is a white-glove experimentation agency. Our senior team brings unique strengths to the table to tackle your most challenging business problems. From server-side to client-side programs, we can work as an extension of your team or with your in-house development resources.
We launch experiments as designed and intended, providing your organization with confidence and continuity.
How we work with you
The world of A/B testing and CRO optimization is ever-changing and can be complicated. As CRO professionals, we offer the most comprehensive knowledge around methodologies and frameworks to offer the most robust experimentation programs. We’re invested in working with you to provide valuable insights into your most challenging business problems.
Members of the Conversion Strategy and Digital Programs teams at work.
Join 10,000 other people who get our newsletter updates