Contact Us

In-house vs. agency experimentation: an honest comparison

Frazer Mawson

There’s one fundamental question that every business leader will have to ask themselves before they can begin to think about conducting their own online, controlled experiments:

‘Should we hire an agency to run our experimentation program or should we build out our own in-house capability?’

This turns out to be a harder question to answer than most people realise.

There are various pros and cons that come with each approach, and what works for one organization won’t necessarily work for another.

This complexity can make it difficult to know which option is right for you, so we decided to offer some guidance in the form of this blog post:

Many of our internal team members here at Conversion have worked in both agency and in-house settings, and in this post, we’re going to pool their experiences to give a completely honest and impartial comparison of the two alternatives.

At the end, we’ll then finish with a quick overview of hybrid experimentation programs and explain how they can sometimes be a better fit for certain organizations – particularly those looking to kick off their in-house programmes or to ramp up their velocity.

Quick disclaimer: we ourselves are an experimentation agency, and as hard as we’ve tried, it can be difficult to get rid of all bias when writing blogs like this. If you feel like we’ve not been completely balanced – or if you think something needs adding – send us an email and if we think your feedback is fair, we’ll be sure to add it into the article!

Contents:

  1. Things to know before we start
  2. The benefits of an agency-led experimentation program
  3. The benefits of an in-house experimentation program
  4. The middle way: hybrid experimentation programs 

Things to know before we start

Before we get into the meat of this post, we thought it worth mentioning a couple of things to give a bit of context to the discussion:

  1. All agencies and in-house teams are different – this post is focussed on understanding how the typical agency experimentation team differs from the typical in-house team, based on the experiences of people who have worked as part of each.
  2. Ultimately, there are pros and cons to each set up, and it’s possible to make a success of either approach. What really matters is that you have a robust methodology and a well-researched strategy in place to guide your experimentation efforts.
  3. Also worth noting: we’ve not bothered to write a complete pros and cons list, because with a comparison like this, the cons of one approach will almost always be the pros of the other. As such, we’ve simply chosen to list the benefits of each, starting with agency-led programs and moving on to in-house programs.

And with that out of the way, let’s get into it…

The benefits of an agency-led experimentation program

1. Higher win rate

A few years ago, Convert.com released a comprehensive report which compared the average win rate of agencies with that of in-house experimentation teams. They included data from more than 28,000 experiments, and found that agencies, on average, had a 21% higher win rate than in-house teams.

Now, of course, win rate isn’t everything.

If we’re running a program for a client and we see that our win rate is above a certain level, we’ll start to question whether the experiments we’ve been running are bold enough – or if maybe we’re being too safe.

One of the biggest advantages of experimentation is that it allows you to trial bold and innovative ideas – ideas that have the potential to revolutionise the way you do business. If your win rate is too high, it might mean that you’re implementing too many safe ideas that you already have good reason to believe will win.

Having said all of this, if win rate is an important consideration for you, then due to the way agency experimentation teams are set up (to be discussed), you may find that this is the route you’d rather go down.

2. Hit the ground running

When you hire an agency, they’ll usually be able to start experimenting on your website straight away.

In fact, when we begin working with a new client, we aim to have experiments up and running within the first two weeks of the project’s start date. This allows us to gather insights and learnings as quickly as possible, and means that we’re able to make an impact right from the get go.

But if you decide to go the in-house route, there’s a lot more that needs to be done before your internal experimentation capability will be fit for purpose.

Not only do you have to go about the process of hiring each individual member of your team – a notoriously difficult task in today’s job market – but you also need to get everything else together: processes, culture, technology & tools, education, etc.

Getting all of this set up can be an extended process, itself requiring a good deal of expertise and experimentation know-how.

But if you hire an agency, this stuff will already be in place, so you can start enjoying a return on your investment almost immediately.

3. Experiment database

Years ago, we decided to start collecting data on every single experiment that we ran.

As a result, we now have an enormous database of experiment results, with in-depth information about a huge range of experiments, conducted for a huge range of clients, operating in a huge range of industries.

This database gives us a real competitive edge when it comes to driving results for our clients.

For example, let’s say that we take on a new client that operates in the financial services sector.

When we’re looking for research to guide the course of our program, we’ll look to our experiment database in search of tactics that have worked for similar clients who operate in financial services.

Of course, there’s no guarantee that what worked for one client will work for another, but at the very least, our probability of success goes up by incorporating this data into our ideation process.

In-house teams, on the other hand, will almost never have access to this kind of data source, which puts them at a competitive disadvantage when it comes to thinking up experiment concepts and building out experiment roadmaps.

4. R&D

Generally speaking, an in-house experimentation expert will spend most of their time researching, ideating, running, and analyzing experiments – with less time available to think about big picture strategy.

But with an agency like ours, we’ve got numerous team members dedicated solely to this stuff. These team members spend time optimizing our processes, refining our testing philosophy, and building our own models so that we can deliver as much value to our clients as possible.

For example, we’ve developed our own neurolabs product, which allows us to leverage cutting edge findings from the field of behavioral science to provide deeper insights into our clients’ web visitors.

In fact, this newly developed product is beginning to play a big part in the work we’re doing for a number of  our clients, including The Times.

Another example of the R&D work we’ve done is our levers framework: as mentioned above, since our inception 14 years ago, we’ve managed to build a huge database of experiment results, and we’ve used this data to develop a systematic theory of the kinds of factors that influence conversion.

This theory – which we refer to as our Levers framework – allows us to make sense of experiment results more easily and helps us devise more impactful experiment programs for our clients.

An overview of our lever’s framework

Our neurolabs product and levers framework are but two examples of many of the R&D work we’re doing, and it’s innovation of this kind that has been a vital ingredient of our success over recent years.

5. Innovation & creativity

Following on from our point about R&D, many of our internal team members (who have previously worked within in-house teams) also make the case that significantly more innovation takes place within agency-led experimentation programs.

Here’s why:

Firstly, any agency-side consultant will be working with multiple clients at any given time.

This means that they’re often exposed to a far wider range of industries, websites, problems, and solutions. They can then take what they’ve learnt with one client and consider how it might apply to other clients they’re working with.

In-house consultants, on the other hand, only work for a single client, usually on a single website. This means that the kind of ‘cross-pollination’ just described rarely if ever occurs, which can lead to ideas growing a bit ‘stale’.

And following on from this point: when agency consultants come up against difficulties, they’re able to draw upon the collective experience, expertise, and creativity of the entire agency to solve their clients’ problems.

Take an agency like ours: at the time of writing, we currently have 16 full-time consultants working here at Conversion. This means that when one of our consultants comes up against a difficult challenge, they’re able to get support and inspiration from a team of 16 skilled consultants, who, themselves, are also working with a diverse group of clients.

In-house experimentation teams very rarely have access to this kind of knowledge/skill pool, which can make it difficult – though, of course, not impossible – for them to compete in terms of innovation.

6. Dedicated resources

Generally speaking, in-house experimentation teams are often set up in a less formal way than agency teams:

There will usually be at least one conversion-focussed specialist, who will work alongside the internal design and development teams to deliver the experimentation program.

But unfortunately, with much of their time taken up with day-to-day work and high-priority business projects, rarely do these in-house designers and developers have much free capacity to support the experimentation team.

This means that in-house consultants are sometimes left pulling their hair out waiting for their experiment concepts to be created.

And it can also mean that, in order to hit their velocity targets, in-house experts are forced to run experiments using highly limiting WYSIWYG editors that only allow them to make relatively minor tweaks to any given page.

With agency teams, on the other hand, the design, development, project management, and QA capability within the agency are all dedicated exclusively to delivering the client’s program. This means that the work often gets completed much more efficiently, and that potentially high-impact experiments are prioritized.

7. Specialisation

One of the challenges with in-house experimentation is that often, due to resource constraints, individuals within the team are forced to do a bit of everything.

For example, many in-house CRO roles are merged with UX roles. This can mean that in-house CRO managers are responsible for running both the experimentation program and the UX optimization program, while also working across website execution for campaigns, looking after core web vitals, and supporting website admins when necessary too. (This was the experience of one of our consultants when working in a previous in-house role)

With agencies, on the other hand, the individuals responsible for delivering your program are allowed to specialise:

This kind of specialisation means that each of these individuals ends up becoming extremely skilled within their limited range of operations – and it is the client, ultimately, who enjoys the benefits of this specialisation.

The benefits of an in-house experimentation program

1. Intimate knowledge of product

When we take on new clients, we take pains to understand them as well as we possibly can: their needs, their expectations, their goals, their products, their services, their websites, their competitors, etc.

This is always one of our first steps, and it gives us a firm understanding on which we can begin running experiments that respond to and attempt to overcome our clients’ core challenges.

But as much time and energy as we dedicate to this stage of our process, it’s difficult for us to understand our clients and their products as well as an in-house expert can.

In-house experts have the luxury of living and breathing your company.

They can dedicate all of their time to your program – rather than having to spread themselves between various clients, as is the case with any agency consultant. They will be able to take part in internal meetings and functions, they will be around for informal conversations, and they will have easier access to your training materials and your internal subject matters experts.

All of this, taken together, means that the in-house expert has a chance to build a deeper, more intuitive understanding of your product and your organization.

If you have an extremely complicated product or service that it will be difficult for an agency consultant to get to grips with, it may make sense for you to go down the in-house route.

2. Lower Cost

If you’re planning to build a one- or two-person experimentation team, you’ll probably find that it costs you a good deal less than would an agency.

Of course, as discussed above, with an agency you get a dedicated designer, developer, project manager, QA engineer, etc., which is where the additional cost comes in.

But if your internal team has the capacity and capability to support your in-house experimentation expert in running the program, then you may find that you have little need for all of the additional resources provided by an agency.

3. Higher testing velocity

In the first section of this post we talked about a report released by Convert in 2019 showing that agency teams had a significantly higher average win rate than in-house teams.

But what the report also showed was that in-house teams had a higher average testing velocity than agency teams.

Testing velocity data taken from convert.com’s 2019 Optimization Maturity Report

Testing velocity is important for a number of reasons. The more tests you can get up and running,

  1. The more data you can gather
  2. The more you can learn
  3. The quicker you can iterate

This is important, and it’s something that we as an agency have spent a lot of time working on ourselves.

But on the other hand – and as we discuss in our blog post on testing velocity – testing velocity is often inversely correlated with the quality of your experiments.

In other words, it’s easy to run more experiments if you cut down on the amount of time you put into research, design, development, and QA, but the impact of those experiments is likely to fall as well.

Considering that agencies, on average, have a higher win rate, it might be fair to draw the tentative conclusion that – again, on average – agencies produce higher quality experiments but in-house teams produce more experiments.

There’s something to be said for each approach, and it’s up to you decide which might work best for your business.

4. Easier to gain stakeholder approval

When we asked our consultants about the pros and cons of in-house experimentation, this was one of the pros that came up more often than any other.

When you bring on an agency to run your experimentation program, there’s only so much they can do to raise awareness about experimentation within your organization. Instead, much of this burden falls upon their points of contact within the business, who aren’t always particularly well qualified to perform this task.

A lack of enthusiasm and support for the program can often mean that it becomes extremely difficult to get approval for experiment concepts, so this is something that it’s important to get right.

As an agency, we’ve had a huge range of experiences with this. Some of our clients have been unbelievably good at championing our results and gaining the support of senior stakeholders, while others have had more difficulty.

But on the other side of the coin, in-house experts have a chance to build real, human relationships with the people who make the decisions within the organization. They have a chance to present their results in internal team meetings, to explain the philosophy behind their experimentation efforts, and to generate enthusiasm for the program.

This means that it’s often easier for internal teams to gain the stakeholder approval they need to make their programs as impactful as possible.

5. Easier to effect cultural change

This point ties in very closely with the one above about stakeholder approval.

An in-house experimentation expert is in a much better position to start building a real culture of experimentation than an agency could ever be.

They have a chance to build close relationships with senior stakeholders, to attend meetings that an agency’s consultant would never have access to, and they have more power and resources available to them to evangelise and build enthusiasm.

If you’re serious about building a real culture of experimentation, at a certain point, you’ll probably need people on the ground, within your organization, to get it done.

6. Ability to get winning experiments implemented

Here’s a scenario that we, as an agency, occasionally come across:

We’ve conducted extensive research to work out why our client’s users aren’t converting in greater numbers.

We’ve ideated, designed, built, and quality assured a variation version of the webpage, which we’ve run against their existing webpage in an a/b test.

The new variation page has produced a strong conversion rate uplift that is likely to generate our clients significant amounts of additional revenue each year.

We’ve then passed the code for this winning variation onto our clients, but for some reason or another, the code is never implemented on their site.

This can happen for a variety of reasons:

Either way, this can be a big problem, completely negating the whole point of experimentation, which is to find out which of your ideas are good and to implement them.

But in theory, this should never occur with in-house teams, because they can work from within the organization to make sure that experimentation is understood and that winning experiments are implemented.

The middle way: hybrid experimentation programs

Hopefully, if we’ve achieved our goal, then you should now know about the important differences between in-house and agency programs, and have a better sense as to the approach that’s likely to work best for you.

This whole discussion can basically be summarised in two key takeaways:

But beyond the agency/in-house binary, there’s also a third option that we’ve not yet talked about: hybrid experimentation.

This approach involves combining the best of both in-house and agency experimentation to achieve the maximum return on investment.

For example, we’ve worked with a number of our clients to help set up their own internal experimentation capability, focussing on things like education, processes, culture, technology, personnel, etc.

This has allowed them to draw upon our experience and expertise – as well as our processes and frameworks – while also reaping many of the benefits that come with in-house experimentation.

And we’ve also worked alongside preexisting in-house teams to support them in developing new, innovative experiment concepts that help drive the maximum possible impact for the program.

The point, then, is this: there’s no one-size-fits-all solution for your experimentation needs. Some organizations may require an agency; some may require an in-house team; and some may require a mixture of the two.

Having read this post, you’ll now hopefully have a better idea as to which set up might work best for you, but if you’d like to talk through your options with an expert, feel free to get in touch. We’re passionate about experimentation, and we’ll do our best to give you an honest assessment of the option that’s best suited to your organization’s needs.

Other articles you might like

Join 5,000 other people who get our newsletter updates