A/B Testing definition

A/B testing, also called "split testing" is a method of improving the performance of some thing by isolating a single element and offering test subjects one of two choices. The two choices differ only in the element being tested. After enough trials to be significant, the results are analyzed to see if there is a difference in the two options that can be attributed to the element being tested. For example, one might test a sales page by using two different headlines and keeping everything else the same. If option A outperforms option B with statistical significance, the difference can be attributed to headline A being more effective. It's also possible to have tests be inconclusive, where any difference can be attributed to random chance.

It's a long, dense paragraph, but it is just one...

John McCabe gave a good definition of A/B testing in his paragraph. What he didn't mention is an issue he and I discussed a while ago on another board. The simplest way to describe this missing issue is sampling error.

When you test group A vs group B you rarely know what each group stands for. Does Group A consist of the first 1000 people who responded to your first message, and Group B consisted of the next 1000 people who responded to your next message? In the real world, that's the way 99.99% of people do A/B testing. That way of doing A/B testing is a waste of time.

You don't know who these people are. Group A just might be the first 1000 people who saw your message on Tuesday morning. But the people who see your message on Tuesday morning might be very different from people who see messages on Tuesday evening, or Saturday afternoon, or Sunday any time of day. In technical terms, you don't know who you are sampling. You might get totally different results if you change the times you send out your messages or the way your message is written or some other variable.

If you think the 2000 people who responded to your A/S testing experiment represent the greater population of potential buyers you are sadly mistaken. The 2000 people who you experiment with are called an "accidental sample" in technical jargon (there are other terms that are used too). It's an accident of time and availability that they happened to be caught in your samples. If you did your A/B testing at another time and with slightly different parameters you might (no, probably will) get totally different results.

The kind of A/B testing most marketers on WF do is like schoolyard baseball compared to major league baseball (or, to use another analogy, they are like the way I fish vs the way John fishes.....I will probably go hungry, while he will probably feast).

Mothers, don't let your kids do A/B testing.

A/B split testing (as the name suggests) is about testing one element versus another. If you have a sales page then you might choose to test two versions of the headline to find which one results in more sales. So you are only testing two versions of one thing at any one time.

The alternative to this is multi-variant split testing where you test many different things on the page at the same time. This is a little more complicated. For those starting out, a/b split testing is usually all that's need to see a big difference in conversions.