CRO GUIDE FOR MARKETING PROFESSIONALS

What is A/B testing?

There are varying complexities to A/B testing and CRO programs. In a nutshell, it’s a method of measuring the performance of different versions of a website or digital experience to determine which is best.

Why should you A/B test and create an experimentation program? (The Benefits of A/B Testing)

To produce desirable results that create more wins and positively impact the budget, structured systems should be implemented.

While the purpose of A/B testing is to gain a better understanding of what converts now, you will be able to derive customer insights from testing to build an improved customer experience for products and services you roll out in the future. Ideally, as your practice matures, you should consider complementary methodologies in your A/B testing program to help you gain broader insights and drive your experimentation program.

Marketers that employ a rigorous experimentation practice experience benefits like reduced bounce rate, increased conversion rate, and higher conversion values. On a more strategic level, you can build a world-class optimization program that will:

• Create more relevant experiences for your customers
• Drive consistent optimization results
• Lift revenue and customer insights

What can you A/B test?

If you’re looking for A/B testing ideas, we suggest you consider testing the homepage, service pages, product landing pages (PLP) and product display pages (PDP) and checkout page of your website or application—think low-hanging fruit.
To determine which pages or experiments to prioritize, refer to the PIE framework developed by Chris Goward.

At Conversion, we use fractional factorial design to ensure that we maximize rewards for running A/B tests.

Good to know:

Recently, our test data suggests that smaller isolated tests (incremental changes) may be just as impactful as running larger-scale tests. However, the mix of the types and sizes of experiments should be purposeful and align with your organization’s goals.

In general, A/B testing and experimentation provides us with answers to key business questions or solutions to customer problems. It’s important to think differently and approach problem-solving in new ways.
When designing your next A/B test, think about including the following elements in your test:

Webpage copy

To run an A/B copy experiment on your website, you can make variations in formatting, positioning, tone, and length to test how your customers respond. Various formats can be tested; for example, you can change the tone of the call to action on your CTAs.

Design and layout

A/B testing is also used in the UX and design practice. By creating variations with layout and design, you can gauge what experience your customers respond to best. Test results can also inform information hierarchy, which can be critical to market your service or product. For example, it can determine the story that you are telling to customers.

Navigation

A/B testing can help determine factors like the best overall navigation experience for your visitors and the most favourable wayfinding paths, i.e. the typical courses users take to find information to help them along their customer journey.
Experiments can reveal a lack of clarity around the primary call-to-action in the navigation, if there was a general lack of urgency, if the menu drop-down was ambiguous or confusing, or if a visitor is ready to spend money, but it’s not clear how to purchase. Testing navigation variants will help you determine the most effective way to design a revenue-driving funnel from all parts of the site.

Forms

Lead form optimization is an art. You will want to examine several aspects to plan your tests accordingly.

You will want to consider testing:

• Form length
• Designing an experience without the use of pop-ups
• Autocomplete
• Single-step vs. multi-step form
• Progress bars or indicators

Be keen to notice user behaviour patterns and design an experience that presents minimal barriers to conversion.

CTAs (Call To Action)

Knowing which CTAs resonate with your customers is a difficult thing to determine—this is where testing comes in handy. Remember, CTAs are much more than just a simple phrase we append onto an offer that we hope will serve as a powerful motivator. Your CTA A/B testing should be clear, concise, and your customers’ should easily understand your unique value proposition. Think about copy variations, placements you can test, and your on-page presentation. Lastly, remember the CTA should also be clear on where it’s going to take users. Too often, we see businesses with CTAs that lack clarity.

Conversion funnels

To move visitors into a sale naturally, you must design a conversion funnel. An optimized conversion funnel can improve your user experience and provide valuable insights about customer behavior. Additionally, conversion funnels will help create a healthy pipeline, contribute to a higher conversion rate, and positively impact your company revenue.

How to design A/B tests

Marketers often use the term ‘A/B testing’ to refer to marketing experimentation in general. But there are multiple ways to structure your experiments. A/B testing is just one of them.
With various frameworks to choose from, selecting how to design your A/B testing can range from very simple to highly complex. Some of the experimentation structures include A/B testing, A/B/n testing, full factorial or multivariate (MVT), and fractional factorial design.

Step 1: Research

Before you enter into any testing program, it is essential to assess the performance of your website or eCommerce platform. Collecting quantitative and qualitative research will help you get a high-level view of your customers’ experience, allow you to design your testing program, and ultimately prepare you to make informed business decisions. Consider analyzing your Google Analytics data, employ a heatmap tool, and conduct customer surveys to start. Allocate an appropriate amount of time to assess and interpret the data.

At Conversion, we’ve developed a proprietary framework called theInfinity Experimentation Process℠ to guide our research. This framework consists of an Explore phase and a Validate phase, with a central area where they meet—this is where you will uncover valuable growth opportunities and insights. We leverage user research, digital analytics, and secondary research from Behavioural Science to inform the next steps in the Explore phase.

Step 2: Observe and formulate the hypothesis

A well-structured hypothesis provides insights whether it is proved, disproved, or results are inconclusive. You should never phrase a marketing hypothesis as a question. It should be written as a statement that can be rejected or confirmed.

“Coming up with something you want to test isn’t hard to do.  Coming up with something you should test can be hard to do.”

— Chris Goward

Further, it should be a statement geared toward revealing insights – with this in mind, it helps to imagine each statement followed by a reason.

Step 3: Create variations

Create variations based on your hypothesis. You can test one variation or several. Ensure you are using an experimentation platform to help you scale your program.

Step 4: Run test

A/B testing principles aren’t as straightforward as they might look. Prior to running your test, you must determine the method you will be employing.

A/B Testing considerations

Once you move beyond the basics of A/B testing, you will notice that tests can increase in complexity. Continued reading: The top 3 mistakes that make your A/B test results invalid

Targeting

If you want to change an element on multiple pages, you could test it across the board with:

• Single page
• Multi-page testing or funnel testing
• Template testing (using the same template but different content)

What are the challenges of A/B testing and CRO?

No matter what experience level you have, there will be A/B testing challenges to navigate.

Deciding what to test

With so many aspects of the website to optimize, how do you know which area to start at first? To help you determine which pages or experiments to prioritize, refer to the PIE framework that Chris Goward developed The PIE Framework.

Formulating hypotheses

Level up by formulating a great hypothesis:How to write a hypothesis for marketing experimentation.

Sample size

The total number of users on your site, your sample size, will determine how quickly your company can test or if your company is in a position to test at this time.

Test design

If your results are disappointing, it may not only be what you are testing – it could be how you are testing. While there are several factors for success, one of the most important to consider is test design or Design of Experiments (DOE). DOE is rooted within statistical theory and defines how experiments are planned to maximize learnings and performance.

Split testing

There are several differences between split testing vs. A/B testing. However, the most critical dissimilarity is that split testing requires you to run two live versions of the site, both hosted on different URLs.

Multivariate (MVT) or Full Factorial Test Design

With multivariate A/B testing, you change several individual sections of the website isolated one against another, by mixing and matching every possible combination.

Fractional factorial test design

Rather than creating a variation for every combination of changed elements (as you would with MVT), you can design your experiment to focus on specific isolations that you hypothesize will have the biggest impact. By building each subsequent variation off of the previous one, you’re able to isolate each change and measure the total impact of all the changes combined maximizing learnings.

Analyzing test results

Even with the best A/B testing setup for optimization, if you aren’t tracking the right metrics to analyze the data, your hard work will go to waste.

Learning from losing experiments

Giving up after seeing losing AB tests is a common reason as to why CRO initiatives fail. If you are consistent, learning and iterating on previous AB tests, you can continue testing until you find the right optimizations.

What are A/B testing tools?

You can find platforms that can help you run A/B testing, multivariate testing and include website personalization capabilities. You will also notice that there are platforms dedicated to improving your collaboration with your team, grow your experimentation program efficiently, and share insights company-wide. Choosing the right tools for your team can be daunting since there are quite a few on the market.

No matter the size or maturity of your experimentation practice, we recommend that you start with a collaboration tool that will help you prioritize hypotheses, store experiment details, comment on tests, and share results and insights with the broader organization.


To begin your search take a look at our collaboration platform, Liftmap.

Are you using the right tools?

The best A/B testing tools are going to help you run tests, make hypotheses and analyze the data.

Choosing the best A/B testing tools for you

As you analyze A/B testing website tools, consider these factors.

• What is the skill level of my team?
• What technical resources will be required?
• What skills will be required to use the software?
• What level of support is Available?
• What volume is needed to run a test?
• Will the software slow down my site?
• How much will the tool cost?
• What are my business objectives?
• What other tools will be needed?

As you progress in the journey, you will need to grow your tech stack.

Tools and business objectives

Deciding what tools and A/B testing software you need will depend on your goals. There are many website A/B testing tools needed to get a full picture; it’s more complex than just running a single tool.


The best A/B testing platform might integrate these objectives.

• Surveys
• User tests
• Business intelligence
• Statistical significance calculator
• Web analytics
• User analytics
• Heatmaps
• A/B testing

Thankfully, there are many big names producing some of the best A/B testing software.

Some examples of A/B testing software

Some A/B testing tools you might have used include Google Analytics, Optimizely, Kameleoon, Adobe Target, Visual Website Optimizer, Unbounce, LeadPages, Kissmetrics or Crazy Egg.

Why you need a team

If you are invested in scaling and maturing your CRO practice, it’s going to take a dedicated team to get you there.

How to scale your A/B testing team

To have a successful experimentation practice, you need to create a testing team of qualified professionals, which may increase your overhead costs.

A well-constructed team consists of the following roles:

• Marketing Strategist
• Data Analyst
• Designer
• Developer
• QA Tech

Anything less than this could lead to disaster and wasted money.

What you can gain from an experimentation platform

With an experimentation platform, you gain scientific results that drive your success. There’s no room for guessing when it comes to your business and goals.

Process

Our team uses scientific experimentation to inform and guide decision-making processes, increasing confidence in key decisions, and reducing business-risk.Learn more about our process.

Team strengths

Conversion is a white-glove experimentation agency. Our senior team brings unique strengths to the table to tackle your most challenging business problems. From server-side to client-side programs, we can work as an extension of your team or with your in-house development resources.
We launch experiments as designed and intended, providing your organization with confidence and continuity.

How we work with you

The world of A/B testing and CRO optimization is ever-changing and can be complicated. As CRO professionals, we offer the most comprehensive knowledge around methodologies and frameworks to offer the most robust experimentation programs. We’re invested in working with you to provide valuable insights into your most challenging business problems.