How to Do A/B Testing: 11 Split Testing Tips

by Brett Farmiloe November 30th, 2021

What is one A/B testing tip to help set up the perfect split test?

To help you set up the perfect split test, we asked marketing professionals and business leaders this question for their insights. From defining success metrics to sticking to a testing schedule, there are several ways to set up your A/B testing for success.

Here are 11 A/B testing tips:

  • Make Specific Goals
  • Define Your KPIs
  • Identify the Right Goals
  • Keep It Simple
  • Define Success Metrics
  • Assure Statistical Significance
  • Test Two Different Pictures
  • Always Be Testing
  • Test One Element at a Time
  • Stick to a Testing Schedule
  • Define the Right User Attributes to Target

Make Specific Goals

I would recommend that you clearly and specifically define your split test metrics beforehand in order to more accurately measure success. Don’t make your goals overly broad or vague; keep them realistic.

What results are you expecting or hoping to see? How would you define a ‘successful’ test? Since A/B testing is an experiment, set a hypothesis you can measure the results against.

Eric Blumenthal, The Print Authority

Define Your KPIs

The first thing you should do is consider what will define success. Knowing the metrics you will base success on will inform how you build and conduct the A/B tests. Establishing a metric of success is key to objectively analyzing your test results, so take care to ensure that your metric is measurable.

Randall Smalley, Cruise America

Identify the Right Goals

In business, it is a well-known fact that you can’t improve what you can’t measure. As such, it is important to set or identify the right goals. A/B testing is a great way to measure the difference in performance on various campaigns, apps, web pages, and even training courses for maximum optimizations. 

By defining what you want to measure and why, you can then pick the right variables that effectively demonstrate any deviations from the control. 

For example, in security awareness training, we can do simple tests like changing the voice or graphic representations within our online course modules to see which version(s) are much more effective, depending on our goal(s).

Nick Santora, Curricula

Keep It Simple

The purpose of an A/B test is to see which of the two scenarios is better. The best tip for setting one up is to keep your variables as small and finite as possible. It's like setting up a science experiment. The more variables there are, the harder it will be to determine which variable affected the change.

For example, if you want to see if changing your CTA button would have an impact on your conversions, you don't want to create a single test where you change the font, text, and color. Instead, create a test for changing the color, another for the text, and another for the font. This way, each data set is unique and actionable.

Phil Strazzulla, Select Software Reviews

Define Success Metrics

Before launching the A/B test, define what metrics will characterize the winner.

Will the winning page be the one that boasts a lower bounce rate? For an e-commerce site, the top page will likely be the one that leads to more conversions. In a B2B setting, it can be the page that generates more qualified leads. Other metrics of success include a higher number of trial sign-ups or fewer abandoned carts.

However you measure success, be sure to define the parameters clearly before starting the test. This way, there's no guesswork as to which page is ideal for your business.

Claire Routh, Markitors

Assure Statistical Significance

Split tests very often fall victim to rushed decisions and test endings, making their results mathematically inconclusive. 

Although it may be quite expensive to achieve high statistical significance such as 99% (meaning there is only a 1% chance that our true/false findings are incorrect), it is worth understanding the connection between the sample size, compared empirical results and their significance. 

With that, we can be aware of how strong the foundations are for our decisions. The test significance can be easily computed with simple online calculators, and its idea may be entirely understood after some reading on the subject.

Michael Sena, SENACEA

Test Two Different Pictures

Do an A/B test with two different pictures to see which one catches people’s attention the most. When it comes to digital marketing, visuals are extremely powerful and usually the first thing people pay attention to. 

So when you’re doing landing pages or social media ads, perhaps you could have one picture with just the product or service and another one with people using the product or service. Either way, remember to make sure that all pictures are high-quality.

Joaquín Roca, Minerva

Always Be Testing

A/B testing should actually stand for 'Always Be Testing' because that's the key to split testing. Just because you found a winner doesn't mean you should stop testing. Implementing a proper testing culture ensures that you're maximizing your content with incremental edits and modifications that can produce significant results over time. 

Whether it's testing the shape, size, color, or text of your CTA button or seeing which type of social proof resonates with your customers, a consistent cycle of A/B testing optimizes your resources and your website while increasing your ROI.

Summer Romasco, Ad Hoc Labs

Test One Element at a Time

One top tip to set up a perfect split test is to test one element at a time. Too many options lead to bad ideas and a waste of time and financial resources. The most common mistake you're likely to commit is to establish an A/B test with multiple variables. 

For instance, in Facebook ads, you'll encounter numerous things to test: ad image, campaign object, target audience, and the list goes on. 

Remember, the fewer variables you have, the faster you'll get relevant test results. However, if you have enough budget, the right expertise, and a broad audience to run more than one A/B experiment all at once, you can freely do it at your own risk.

Jake Smith, Absolute Reg

Stick to a Testing Schedule

A/B testing is an incredible tool for online businesses, but it can consume an enormous amount of time without proper scheduling. The bigger your company, the more requests or opportunities you will have to test variables.

If you don't maintain a set schedule, then you run a serious risk of not testing long enough or being bogged down in planning and checking current tests.

Our advice is to stick to a schedule, like only launching tests on Monday, reviewing test requests once per month, etc. This will not only help you stay organized but also help you maintain your sanity!

Sylvia Kang, Mira

Define the Right User Attributes to Target

Defining a specific set of customers to target will help you determine what changes to make to your testing variants to create an experience tailored to their needs. For example, analyzing your website data will enable you to define user attributes like device category or model. 

You can then create a targeted testing variant on your landing page and serve it to website visitors who share those attributes. Usability is critical for the success of any website. And the results of a properly targeted A/B test will better show the impact of your experiment.

Jared Pobre, Caldera + Lab

Terkel creates community-driven content featuring expert insights. Sign up at to answer questions and get published. 


Ready to start building?

Your cart