Author: Frazer Mawson

There’s one fundamental question that every business leader will have to ask themselves before they can begin to think about conducting their own online, controlled experiments:  

‘Should we hire an agency to run our experimentation programme or should we build out our own in-house capability?’

This turns out to be a harder question to answer than most people realise.  

There are various pros and cons that come with each approach, and what works for one organisation won’t necessarily work for another.  

This complexity can make it difficult to know which option is right for you, so we decided to offer some guidance in the form of this blog post:

Many of our internal team members here at Conversion have worked in both agency and in-house settings, and in this post, we’re going to pool their experiences to give a completely honest and impartial comparison of the two alternatives. 

At the end, we’ll then finish with a quick overview of hybrid experimentation programmes and explain how they can sometimes be a better fit for certain organisations – particularly those at the start of their experimentation journey. 

Quick disclaimer: we ourselves are an experimentation agency, and as hard as we’ve tried, it can be difficult to get rid of all bias when writing blogs like this. If you feel like we’ve not been completely balanced – or if you think something needs adding – send us an email and if we think your feedback is fair, we’ll be sure to add it into the article!

Contents:

1. Things to know before we start
2. The benefits of an agency-led experimentation programme
   • Higher win rate
   • Hit the ground running
   • Experiment database
   • Innovation
   • Dedicated resources
   • Specialisation
3. The benefits of an in-house experimentation programme
   • Intimate knowledge of product
   • Lower cost
   • Higher testing velocity
   • Easier to win stakeholder approval
   • Easier to effect cultural change
   • Ability to get winning experiments implemented
4. The middle way: hybrid experimentation programmes 

Things to know before we start

Before we get into the meat of this post, we thought it worth mentioning a couple of things to give a bit of context to the discussion: 

1. All agencies and in-house teams are different – this post is focussed on understanding how the typical agency experimentation team differs from the typical in-house team, based on the experiences of people who have worked as part of each. 
2. Ultimately, there are pros and cons to each set up, and it’s possible to make a success of either approach. What really matters is that you have a robust methodology and a well-researched strategy in place to guide your experimentation efforts. 
3. Also worth noting: we’ve not bothered to write a complete pros and cons list, because with a comparison like this, the cons of one approach will almost always be the pros of the other. As such, we’ve simply chosen to list the benefits of each, starting with agency-led programmes and moving on to in-house programmes.

And with that out of the way, let’s get into it…

The benefits of an agency-led experimentation programme

1. Higher win rate

A few years ago, Convert.com released a comprehensive report which compared the average win rate of agencies with that of in-house experimentation teams. They included data from more than 28,000 experiments, and found that agencies, on average, had a 21% higher win rate than in-house teams. 

Now, of course, win rate isn’t everything.

If we’re running a programme for a client and we see that our win rate is above a certain level, we’ll start to question whether the experiments we’ve been running are bold enough – or if maybe we’re being too safe. 

One of the biggest advantages of experimentation is that it allows you to trial bold and innovative ideas – ideas that have the potential to revolutionise the way you do business. If your win rate is too high, it might mean that you’re implementing too many safe ideas that you already have good reason to believe will win. 

Having said all of this, if win rate is an important consideration for you, then due to the way agency experimentation teams are set up (to be discussed), you may find that this is the route you’d rather go down.  

2. Hit the ground running

When you hire an agency, they’ll usually be able to start experimenting on your website straight away. 

In fact, when we begin working with a new client, we aim to have experiments up and running within the first two weeks of the project’s start date. This allows us to gather insights and learnings as quickly as possible, and means that we’re able to make an impact right from the get go. 

But if you decide to go the in-house route, there’s a lot more that needs to be done before your internal experimentation capability will be fit for purpose. 

Not only do you have to go about the process of hiring each individual member of your team – a notoriously difficult task in today’s job market – but you also need to get everything else together: processes, culture, technology & tools, education, etc.

Getting all of this set up can be an extended process, itself requiring a good deal of expertise and experimentation know-how. 

But if you hire an agency, this stuff will already be in place, so you can start enjoying a return on your investment almost immediately.  

3. Experiment database

Years ago, we decided to start collecting data on every single experiment that we ran. 

As a result, we now have an enormous database of experiment results, with in-depth information about a huge range of experiments, conducted for a huge range of clients, operating in a huge range of industries. 

This database gives us a real competitive edge when it comes to driving results for our clients.  

For example, let’s say that we take on a new client that operates in the financial services sector. 

When we’re looking for research to guide the course of our programme, we’ll look to our experiment database in search of tactics that have worked for similar clients who operate in financial services. 

Of course, there’s no guarantee that what worked for one client will work for another, but at the very least, our probability of success goes up by incorporating this data into our ideation process. 

In-house teams, on the other hand, will almost never have access to this kind of data source, which puts them at a competitive disadvantage when it comes to thinking up experiment concepts and building out experiment roadmaps. 

4. R&D

Generally speaking, an in-house experimentation expert will spend most of their time researching, ideating, running, and analysing experiments – with less time available to think about big picture strategy. 

But with an agency like ours, we’ve got numerous team members dedicated solely to this stuff. These team members spend time optimising our processes, refining our testing philosophy, and building our own models so that we can deliver as much value to our clients as possible. 

For example, we’ve developed our own neurolabs product, which allows us to leverage cutting edge findings from the field of behavioural science to provide deeper insights into our clients’ web visitors. 

In fact, this newly developed product is beginning to play a big part in the work we’re doing for a number of  our clients, including The Times. 

Another example of the R&D work we’ve done is our levers framework: as mentioned above, since our inception 14 years ago, we’ve managed to build a huge database of experiment results, and we’ve used this data to develop a systematic theory of the kinds of factors that influence conversion. 

This theory – which we refer to as our Levers framework – allows us to make sense of experiment results more easily and helps us devise more impactful experiment programmes for our clients. 

Our neurolabs product and levers framework are but two examples of many of the R&D work we’re doing, and it’s innovation of this kind that has been a vital ingredient of our success over recent years. 

5. Innovation & creativity

Following on from our point about R&D, many of our internal team members (who have previously worked within in-house teams) also make the case that significantly more innovation takes place within agency-led experimentation programmes. 

Here’s why: 

Firstly, any agency-side consultant will be working with multiple clients at any given time.

This means that they’re often exposed to a far wider range of industries, websites, problems, and solutions. They can then take what they’ve learnt with one client and consider how it might apply to other clients they’re working with. 

In-house consultants, on the other hand, only work for a single client, usually on a single website. This means that the kind of ‘cross-pollination’ just described rarely if ever occurs, which can lead to ideas growing a bit ‘stale’. 

And following on from this point: when agency consultants come up against difficulties, they’re able to draw upon the collective experience, expertise, and creativity of the entire agency to solve their clients’ problems. 

Take an agency like ours: at the time of writing, we currently have 16 full-time consultants working here at Conversion. This means that when one of our consultants comes up against a difficult challenge, they’re able to get support and inspiration from a team of 16 skilled consultants, who, themselves, are also working with a diverse group of clients.

In-house experimentation teams very rarely have access to this kind of knowledge/skill pool, which can make it difficult – though, of course, not impossible – for them to compete in terms of innovation.  

6. Dedicated resources

Generally speaking, in-house experimentation teams are often set up in a less formal way than agency teams:

There will usually be at least one conversion-focussed specialist, who will work alongside the internal design and development teams to deliver the experimentation programme. 

But unfortunately, with much of their time taken up with day-to-day work and high-priority business projects, rarely do these in-house designers and developers have much free capacity to support the experimentation team. 

This means that in-house consultants are sometimes left pulling their hair out waiting for their experiment concepts to be created. 

And it can also mean that, in order to hit their velocity targets, in-house experts are forced to run experiments using highly limiting WYSIWYG editors that only allow them to make relatively minor tweaks to any given page.

With agency teams, on the other hand, the design, development, project management, and QA capability within the agency are all dedicated exclusively to delivering the client’s programme. This means that the work often gets completed much more efficiently, and that potentially high-impact experiments are prioritised.

7. Specialisation

One of the challenges with in-house experimentation is that often, due to resource constraints, individuals within the team are forced to do a bit of everything. 

For example, many in-house CRO roles are merged with UX roles. This can mean that in-house CRO managers are responsible for running both the experimentation programme and the UX optimisation programme, while also working across website execution for campaigns, looking after core web vitals, and supporting website admins when necessary too. (This was the experience of one of our consultants when working in a previous in-house role)

With agencies, on the other hand, the individuals responsible for delivering your programme are allowed to specialise:

• Consultants are able to focus exclusively on strategy, without having to spend their time project managing the programme or chipping in on design/development/QA work. 
• Designers and developers are able to learn and hone the specific skills required for high-level experimentation – skills which are often subtly but significantly different from those of most typical design and development roles. 
• Top agencies will also often have a dedicated QA team, whose primary responsibility will be to ensure that your experiments meet certain standards. 

This kind of specialisation means that each of these individuals ends up becoming extremely skilled within their limited range of operations – and it is the client, ultimately, who enjoys the benefits of this specialisation. 

The benefits of an in-house experimentation programme

1. Intimate knowledge of product

When we take on new clients, we take pains to understand them as well as we possibly can: their needs, their expectations, their goals, their products, their services, their websites, their competitors, etc. 

This is always one of our first steps, and it gives us a firm understanding on which we can begin running experiments that respond to and attempt to overcome our clients’ core challenges. 

But as much time and energy as we dedicate to this stage of our process, it’s difficult for us to understand our clients and their products as well as an in-house expert can. 

In-house experts have the luxury of living and breathing your company. 

They can dedicate all of their time to your programme – rather than having to spread themselves between various clients, as is the case with any agency consultant. They will be able to take part in internal meetings and functions, they will be around for informal conversations, and they will have easier access to your training materials and your internal subject matters experts. 

All of this, taken together, means that the in-house expert has a chance to build a deeper, more intuitive understanding of your product and your organisation. 

If you have an extremely complicated product or service that it will be difficult for an agency consultant to get to grips with, it may make sense for you to go down the in-house route. 

2. Lower Cost

If you’re planning to build a one- or two-person experimentation team, you’ll probably find that it costs you a good deal less than would an agency. 

Of course, as discussed above, with an agency you get a dedicated designer, developer, project manager, QA engineer, etc., which is where the additional cost comes in. 

But if your internal team has the capacity and capability to support your in-house experimentation expert in running the programme, then you may find that you have little need for all of the additional resources provided by an agency. 

3. Higher testing velocity

In the first section of this post we talked about a report released by Convert in 2019 showing that agency teams had a significantly higher average win rate than in-house teams.  

But what the report also showed was that in-house teams had a higher average testing velocity than agency teams. 

Testing velocity is important for a number of reasons. The more tests you can get up and running,

1. The more data you can gather
2. The more you can learn
3. The quicker you can iterate

This is important, and it’s something that we as an agency have spent a lot of time working on ourselves.

But on the other hand – and as we discuss in our blog post on testing velocity – testing velocity is often inversely correlated with the quality of your experiments.

In other words, it’s easy to run more experiments if you cut down on the amount of time you put into research, design, development, and QA, but the impact of those experiments is likely to fall as well. 

Considering that agencies, on average, have a higher win rate, it might be fair to draw the tentative conclusion that – again, on average – agencies produce higher quality experiments but in-house teams produce more experiments. 

There’s something to be said for each approach, and it’s up to you decide which might work best for your business. 

• Read an interesting Q&A about how AT&T increased the impactfulness of their experimentation programme by reducing their testing velocity by 40%. 

4. Easier to gain stakeholder approval

When we asked our consultants about the pros and cons of in-house experimentation, this was one of the pros that came up more often than any other.

When you bring on an agency to run your experimentation programme, there’s only so much they can do to raise awareness about experimentation within your organisation. Instead, much of this burden falls upon their points of contact within the business, who aren’t always particularly well qualified to perform this task. 

A lack of enthusiasm and support for the programme can often mean that it becomes extremely difficult to get approval for experiment concepts, so this is something that it’s important to get right. 

As an agency, we’ve had a huge range of experiences with this. Some of our clients have been unbelievably good at championing our results and gaining the support of senior stakeholders, while others have had more difficulty.

But on the other side of the coin, in-house experts have a chance to build real, human relationships with the people who make the decisions within the organisation. They have a chance to present their results in internal team meetings, to explain the philosophy behind their experimentation efforts, and to generate enthusiasm for the programme. 

This means that it’s often easier for internal teams to gain the stakeholder approval they need to make their programmes as impactful as possible. 

5. Easier to effect cultural change

This point ties in very closely with the one above about stakeholder approval.

An in-house experimentation expert is in a much better position to start building a real culture of experimentation than an agency could ever be. 

They have a chance to build close relationships with senior stakeholders, to attend meetings that an agency’s consultant would never have access to, and they have more power and resources available to them to evangelise and build enthusiasm. 

If you’re serious about building a real culture of experimentation, at a certain point, you’ll probably need people on the ground, within your organisation, to get it done. 

6. Ability to get winning experiments implemented

Here’s a scenario that we, as an agency, occasionally come across:

We’ve conducted extensive research to work out why our client’s users aren’t converting in greater numbers. 

We’ve ideated, designed, built, and quality assured a variation version of the webpage, which we’ve run against their existing webpage in an a/b test. 

The new variation page has produced a strong conversion rate uplift that is likely to generate our clients significant amounts of additional revenue each year.

We’ve then passed the code for this winning variation onto our clients, but for some reason or another, the code is never implemented on their site. 

This can happen for a variety of reasons: 

• Maybe their developers have no free capacity to implement the code
• Maybe their senior stakeholders are unwilling to give final sign off
• Maybe there’s nobody within the business project managing the implementation. 

Either way, this can be a big problem, completely negating the whole point of experimentation, which is to find out which of your ideas are good and to implement them. 

But in theory, this should never occur with in-house teams, because they can work from within the organisation to make sure that experimentation is understood and that winning experiments are implemented. 

The middle way: hybrid experimentation programmes

Hopefully, if we’ve achieved our goal, then you should now know about the important differences between in-house and agency programmes, and have a better sense as to the approach that’s likely to work best for you. 

This whole discussion can basically be summarised in two key takeaways:

• As a result of increased specialisation and dedicated resources to experimentation, agency teams are often able to be more efficient and innovative, which allows them to achieve better win rates on average.
• In-house teams can be cheaper to run in the long-run and they come with a higher testing velocity. They’re also often more able to gain strong stakeholder approval for the programme and to impact the culture within the business. 

But beyond the agency/in-house binary, there’s also a third option that we’ve not yet talked about: hybrid experimentation. 

This approach involves combining the best of both in-house and agency experimentation to achieve the maximum return on investment.   

For example, we’ve worked with a number of our clients to help set up their own internal experimentation capability, focussing on things like education, processes, culture, technology, personnel, etc. 

This has allowed them to draw upon our experience and expertise – as well as our processes and frameworks – while also reaping many of the benefits that come with in-house experimentation. 

And we’ve also worked alongside preexisting in-house teams to support them in developing new, innovative experiment concepts that help drive the maximum possible impact for the programme. 

The point, then, is this: there’s no one-size-fits-all solution for your experimentation needs. Some organisations may require an agency; some may require an in-house team; and some may require a mixture of the two.  

Having read this post, you’ll now hopefully have a better idea as to which set up might work best for you, but if you’d like to talk through your options with an expert, feel free to get in touch. We’re passionate about experimentation, and we’ll do our best to give you an honest assessment of the option that’s best suited to your organisation’s needs.

As a battle-hardened experimentation agency with more than fourteen years experience running advanced experimentation programmes for our clients, we here at Conversion have spent a lot of time thinking about how we can make our programmes as impactful as possible.

As a result of all this thinking, we’ve identified three key factors (to be discussed in the next section) that we believe determine the success or failure of any given experimentation programme. 

In this post, we’re going to zone in on one of these factors – specifically, testing velocity – by explaining what it is and why it’s important. With the basics covered, we’ll then go on to share some of the strategies that we’ve used to increase our agency-wide testing velocity by more than 30%.

So, in summary, throughout this post we’re going to cover:

1. What is testing velocity?
2. Why is testing velocity important?
3. How to increase your testing velocity

What is testing velocity?

Testing velocity is a measure of how quickly you can get your experiments up and running, from research and ideation through to launch. 

It’s one of the three core factors that determine the success of any experimentation programme – the other two are testing volume, which is the number of experiments you’re able to run in any given time period, and testing value, which is a calculation of your win-rate multiplied by your average conversion uplift. 

So, in essence, the more experiments you can run (volume), the quicker you can launch them (velocity), and the better those experiments are (value), the greater your ROI will be. Or,

One important thing to understand about these factors is that they’re all closely related. That means that when you work to improve one of them, you’re also likely to impact one or both of the others as well. 

For example, let’s say that you decide to improve your testing velocity by cutting down the time you spend on user research. All things being equal, you may find that this change has a negative impact on your win rate, and that this reduced win rate cancels out any of the improvements that came from increasing your testing velocity. 

This example illustrates an important point: when you take measures to increase your testing velocity, you need to make absolutely certain that these measures aren’t negatively affecting either of the other two factors.

All of the strategies we discuss in the final part of this blog post have been designed with this in mind. 

Why is testing velocity important?

On a big picture level, testing velocity is important because it impacts your ROI. 

But why’s that so?

Well, generally speaking, the quicker you can take your experiments from ideation to launch:

• the more information you can gather about your users 
• the greater the number of challenger variations you’ll be able to test in any given time period
• the greater the number of hypotheses you’ll be able to validate
• the greater the impact you’ll be able to produce on your sitewide conversion rate

6 strategies to increase your testing velocity

When trying to improve your testing velocity, the goal should always be to increase your launch speed while at the same time ensuring that your experiment quality – or, what we refer to as ‘value’ – stays high. 

The strategies we’re going to unveil in this section have allowed us to do exactly that.

In fact, since 2018 we’ve improved our average launch speed by more than 32%.

Also worth noting is that we managed to achieve and maintain this uplift during a time when our agency more than doubled in size.

Read on to find out how we did it.

1. Data gathering and reporting

Four years ago, we made a conscious effort as an agency to increase our testing velocity.

Unfortunately, what we quickly realised was that, up until then, we hadn’t been gathering much data on the amount of time it took us to go from strategy to launch.

This made it extremely difficult for us to increase our speed, because without some baseline level to compare our progress to, how could we know if we were improving?

As a result, the first thing we did to increase our testing velocity was to actually start tracking it. 

This gave us a strong foundation on which we could begin looking to optimise. But data gathering alone isn’t enough to move the needle – you also need to start reporting on your data too. 

Reporting gives you a chance to keep your goals fresh in your mind, to consciously monitor your progress, and to narrow your focus on those changes that are actually going to make a difference.

Soon after we started tracking our testing velocity three years ago, we also started reporting on it internally. Since then, we’ve developed a number of reporting mechanisms – primarily using google data studio – that we’ve used to display and keep track of our testing velocity. 

Below is an example of one of the first dashboards we put together to report on our velocity. It contains the number and percentage of experiments that we launched in less than three weeks, and the number and percentage that we launched in less than two. 

This kind of reporting setup is a big part of the reason behind our dramatic testing velocity improvement over the last few years.

2. Build smaller experiments

‘The bigger the build, the bigger the uplift,’ is a truism within the world of experimentation, based on the idea that the bigger and more complex the changes you make to a web page, the bigger your conversion rate uplift is likely to be. 

But is this true?

A while ago, we dug into our database of experiments to look at the average uplift for experiments of different sizes. What we found was that large experiments were no more likely to win than minor tweaks – and that they actually had a slightly smaller uplift than tweaks as well (see chart below)!

Of course, there may be times when it becomes necessary to spend a good deal of time building out an elaborate experiment: the key to optimising your testing velocity is working out when this kind of additional time and energy is justified and when it’s simply slowing your process down.

Smaller experiments typically require less time spent on research, ideation, design, and development, which means that you can run more of them over a shorter period of time. 

This is why we always try to begin our experimentation programmes with a minimum viable experiment (MVE), which we define as the smallest possible experiment that will allow us to validate our hypothesis. 

Generally speaking, we will only invest in experiments that have a large build time if we’ve proven, either through a smaller experiment (possibly an MVE) or comprehensive user research, that the increased build time will be worthwhile. 

This approach allows us to start testing straight away, at the outset of a new project, without needing to wait weeks or even months for research, design, and development to come together. 

Ultimately, this gives us a chance to start impacting conversion rates immediately, while also gathering insights about our clients’ users which we can use to inform future experiment iterations. 

This approach is a large part of the reason why we’re so often able to achieve a positive return on investment for our clients within the first 12 weeks of their programmes. 

3. Avoid deep-dive analysis (when appropriate)

Having just completed an experiment, it can sometimes be tempting to spend weeks delving through your results, analysing various non-primary metrics in the hopes of uncovering insights that will unlock the rest of your experimentation programme.

Unfortunately, this almost never happens, which means that this additional analysis, while interesting, turns out to be mostly pointless. 

Instead, we recommend asking yourself the following question: how often do I find that the decision for my next experiment (the iteration) is made based on the analysis of a non-primary metric?

If your answer is ‘rarely’ or ‘never,’ then you might want to rethink the way you do your analysis. 

Our approach often involves focussing solely on our primary KPIs and only digging into non-primary metrics when we wish to gain deeper insights into the ‘why’ behind the experiment result.  

This approach means that we don’t waste time producing analyses that ultimately have no impact on the success of our programmes. 

4. Experiment internally

When attempting to work out which parts of your process are essential and which can be cut down, you’ll probably encounter a number of conflicting opinions amongst different members of your team.

Some may feel that extensive research is necessary for every single experiment, no matter how small the intended changes. Others may believe that research can be reduced for certain kinds of experiments, but that your usual design process should always be followed to protect the quality of your experiments. 

How do you decide between these two well-reasoned perspectives?

The same way you decide between two well-designed web pages: you run a series of experiments on them and see which one works better. 

Experimentation needn’t be limited to the conversion rates on your website – it can be applied to every activity within your organisation, from your R&D and product development right on through to your internal processes and procedures. 

As an experimentation agency that champions the power of experimentation, we’re constantly running tests on our internal processes, and this has allowed us to significantly improve our testing velocity while maintaining the quality of our experiments. 

5. Automation

We’ve recently launched our own R&D department, which is responsible, among other things, for automating many of the administrative tasks that our internal team does on a daily basis.

So far, this has allowed us to avoid many duplications of work while also cutting down on the time we spend doing menial tasks. As a result, we now have more time to focus on things that matter – like developing the best, most impactful experimentation programmes for our clients.

Most of the automation we’ve done so far has been relatively minor, focusing on small, incremental changes, but by cutting out fifteen minutes of work here and half an hour there for every single experiment we run, these changes are starting to add up.

Consequently, automation is beginning to play a bigger and bigger role in our ability to increase our testing velocity. 

6. Gain stakeholder approval early

Whether you’re working in-house or agency side, there’s every likelihood that each of your experiments will need to be signed off by at least one – usually several – stakeholders before it can go live. 

Frustrating as this can be at times, it needn’t in and of itself negatively impact your testing velocity. 

Problems begin to arise, however, when experiments are rejected at the later stages of production. When this happens, it can mean that tens of hours have been sunk into an experiment that will never see the light of day. Not only is this a terrible waste of your resources, but it also badly hurts your testing velocity. 

To avoid this outcome, we always do everything within our power to gain stakeholder approval as early on in our process as we can. If an experiment’s going to be rejected, we want it to be rejected at the ideation stage. If this isn’t possible, then at the very least we want it to happen at the design stage, before hours of development have been put into it. 

There will always be experiments that you design and build out only to find that your stakeholders aren’t happy with the final implementation – the goal here is to make sure that this happens as rarely as possible. 

Final Thoughts

Focussing on improving your testing velocity is one of the best things you can do to increase the ROI of your experimentation programme. For us, it’s been a gradual process that continues to this day – and the results have been extremely worthwhile, allowing us to provide more value to our clients than ever before. 

If you’re serious about improving the testing velocity of your programme, the strategies outlined in this post offer a good place to start.