5 Common A/B Testing Problems and How to Overcome Them

A/B testing is easy enough to get started, but as your testing process progresses, you’ll probably want more innovative options.

And that’s when problems arise and the process gets tricky.

According to Adobe, 82% of digital marketing respondents say that knowing how to test effectively “poses a challenge for their organizations.”

That’s a lot of people who may not know how to execute an A/B test experiment at all, let alone the correct one.

And figuring out all of that could be a big task, especially if those people still try to run an experiment and simply hope it will benefit them without knowing what they’re doing in the first place.

Complex A/B Testing are complex because of ab test development challenges, such as flashing of original content, alignment, overwriting with code and integrations, and much more. Resolving them is necessary to deliver a clean A/B test without any loopholes.

To avoid running ineffective tests that are a waste of time and money, there are experts you can hire to run testing for you. Good A/B testing may be all you need, and there are people out there skilled enough to run a variety of tests on your behalf. But there is a catch: It can be hard to reach out and hire help, especially when many outsourced A/B testing agencies have a very narrow list of tests they can deliver.

What if you need a more complicated test than an agency offers? And if a test is that difficult, should you give up on that idea?

Absolutely not. Top performers in this industry, such as Google, Amazon, and Facebook, run hundreds of experiments per month. As of 2017, Amazon, Booking.com, Google, and Facebook were each conducting more than 10,000 online controlled experiments per year. They are the industry leaders who come up with new ways to interact with and engage their users.

Our clients — usually website owners or conversion rate optimization (CRO) experts — often come up with crazy ideas for testing, backed up with logical hypotheses. And even though their testing ideas could involve some complex setup, it’s our job to validate their hypotheses by delivering the desired experiments.

There are plenty of arguments to be made that, if your A/B testing idea sounds too crazy, you should just give up on it, filing it away as “not worth it” without any better reason than because the premise is complicated.

We at BrillMark don’t think that’s necessary.

In fact, we can’t imagine a worse idea than neglecting CRO efforts just because a test seems too complicated to build. Hire A/B Test Developers to start your A/B Testing professionally.

To prove it, here are a few information-rich examples — and a little basic logic — to help prove the following testing fallacies wrong:

5 Testing Fallacies You Should Ignore When Running A/B Tests

1. A/B Testing is not for a small number of users.

How to develop an ab test in a website with low traffic is a very popular A/B testing problem.

There is no rule of thumb for how many visitors are needed to start with A/B testing, and every website starts with nothing. Your site’s traffic could be earned via marketing but not converting. And yes, it’s sometimes difficult to reach a conclusion with A/B testing on a site that doesn’t have a ton of traffic.

However, you only need enough traffic to reach statistical significance with your tests. For a low-traffic website, it’s just as — or even more — important to provide a better user experience to promote consistency and growth. It is okay to have experiments going before you have enough people to have at least 95% confidence in the end results.

Optimizely’s “Testing tips for low-traffic sites” has 8 tips for maximizing testing with small sample sizes, and here are a few of our favorites:

1. Focus on changes with big impact.

For example, Leonardo, a service site which helps the hotels maintain their digital experience online, shares this relevant example about how they created a big change in appearance through a banner only.

Source: (leonardo.com)

This is a high-impact change. These kinds of visible changes are useful for low traffic sites to capture the reaction of their users via A/B testing and to maintain consistency. As it is all about getting bigger with your audience.

2. Test with the intention of improving SEO and user experience.

A/B testing can be done specifically to enhance traffic. Many online businesses test the efficacy of their PPC ads on a regular basis, and that testing may include variations in CTAs and ad messaging.

3. Skip the multivariate testing until you have more traffic.

Multivariate testing includes multiple variables to be tested at one time. To know which combination of elements looks the best, it will take longer to get significant results. Results will be measurable faster if less variables are involved.

There are many A/B testing sample size tools out there, which can evaluate whether testing will be worth it or not for your site.

If your sample still looks very small but you want to run a test anyway, you can use Google Optimize, which does not have minimum traffic conditions yet.

2. It’s not worth running tests that take a long time to show results.

We disagree. An experiment is not bounded by the time it takes to do its magic. Some ab tests are meant to be conducted with patience, as some changes or elements could be too advanced to reflect an impact on your audience immediately.

Based on your priorities, you can choose not to run a test that seems too time consuming. But if you have a brilliant idea for a test that takes weeks to show results, do you really want to give up the information a test like that could provide? Giving it up entirely means never getting those results.

Hackernoon has a blog post explaining more ways to overcome A/B testing mistakes. The following graph is composed of data from a survey of A/B testing practitioners regarding how long they think an individual A/B test usually lasts:

Source: hackernoon

One year is obviously not a popular amount of time to run an A/B test for, but “One month” ranked highest, and “Multiples months” was not uncommon. This shows that plenty of A/B testers think it’s totally normal for a test to take longer than a week to deliver significant results.

The entire point of A/B testing may be to save your website from displaying fancy elements which look best in theory but which turn out to negatively affect the business.

The only benefit you have by not performing a long-term A/B test is never finding out if your changes will negatively affect your website and conversions.

Is that really the “benefit” you want?

3. A/B testing isn’t worth doing if it’s built on popular beliefs and delusions.

Tests are always worth doing if the hypotheses behind them are generated with constructive knowledge about consumer behavior and analytics. But even if they’re not built on that knowledge, A/B tests are the safety nets to save you from falling into the trap of making decisions based on guesses or assumptions.

Many people who have no understanding of CRO best practices still believe their own ideas about their websites, and they assume those ideas will definitely work. Especially in those cases, A/B testing works as a mirror to show how profitably wrong they could be.

In The Netflix Tech blog, Netflix explains the importance and process of selecting the covers for their videos through A/B test experiments.

Source: The Netflix Tech Blog

The two marked covers are the ones which outperformed others in impressions.

If looked without judgement they all look cool and interesting, but the reason for a video cover’s existence is to grab the user’s attention and gain a view. A/B testing enables you to know that, know what is more appealing to the user.

Excellent A/B testing helps us give up our assumptions by letting us explore what the consumer actually wants, not what we think they want. Testing can shift those ideas from featuring an owner or manager’s personal tastes or other common beliefs into including your customer’s preferences and buying habits.

There is nothing wrong with someone having confidence in their own ideas. They may or may not work, but you won’t know it unless you test them.

4. It is too difficult to get a ‘certain idea’ tested properly.

Unlike simple A/B tests that may be run to discount personal ideas about website performance, the opposite problem can also derail testing process: When you know a lot about CRO best practices, you may assume it will be difficult or even impossible to build and run a complex A/B test.

For example, in single-page applications in which the page’s elements gets updated on the basis of actions, it can be tough to run an experiment because the dynamic content overwrites the changes made via the test. One has to have enough knowledge about all the possible ways to overcome that.

Luckily for our clients, we know just how to do that.

Don’t give up on your complex ab testing ideas just because they seem too difficult to create and run. In the tech world, there are often various A/B testing solutions to every problem; if you can’t handle building a test at the moment, pass it over to someone who can. We’d love to help your site reach its potential, and you won’t have to give up on your complex testing goals.

5. This change was going to happen regardless of the A/B test result.

Some companies redesign their entire website on a set period of intervals. This provides excuses such as, “Well, this is what they have been planning for a long time, so why test if the application change is already confirmed?”

For starters, because nothing is as perfect as it may seem.

For example, if you are a very large ecommerce company with revenue in the millions, wouldn’t you care if any new change which was set in stone could cost you money? Because, in optimization, a positive or negative change can increase or decrease orders by 10%.

Of course, the person or department who proposed the scheduled change may tell you it’s set up to provide a positive change. But is that person accountable for the site performing better in terms of profit, or are you?

At the end of the day, the user decides the future of your business, and the design has to be in favor of them having a better user experience. Without A/B testing to know, a better UX may be impossible.

The Real Value of Overcoming A/B Testing Problems

Companies that want to grow and adapt are running tests every day. Updates and trends change day by day; some are easy to adapt to, and some aren’t. But instead of following the herd, it’s better to go with a practical approach, even if the path gets rocky.

That includes following your customer’s minds instead of being a “Yes-Man” to someone else’s opinions.

There may come restrictions, issues, and difficulties with bringing a testing idea to reality. But there are more than enough ways to overcome every single one of them. And we at BrillMark can help you with that.