Potential weakness in split testing?

[DavidO]

I always thought split-testing was pretty straightforward. I understand that it takes about 1000 unique visitors to get conclusive results. I can get this amount of traffic in one and a half days on average.

But how do you account for the fact that many (maybe most?) customers don't buy on the first visit. I talk to a lot of my customers and they often tell me it takes them two weeks or more to make the decision to buy. During that time they visit my site any number of times (it's not just a sales page, I have a lot of useful content).

So how do you account for this when split testing? Say I'm testing a new headline. The person who buys during that testing period may have first visited my site two weeks previously. It may have been the previous headline and copy that really made the impression on this customer, not the new one.

Or, visitors on the day of testing may be really hooked by my new headline but they wait a week to buy, long after I've dumped the "failed" headline.

This factor is not going to make a big difference with squeeze pages or simple sales pages where people tend to make instant decisions but I think it could make a helluva difference with sites like mine.

I'm sure there are ways to account for this, but I can't see how at the moment. I would say the best rule is to test major elements for at least two weeks at a time, maybe even longer.

Am I missing something?

[Chris Ramsey]

If you're using a good testing app (say Google Website Optimizer) the same visitors won't be counted more than once and they will see the same ad each and every time.

This way you'll know exactly which ad converts better, no matter how many times they come back to the site.

[Paul Hancox]

Yes, time is an important element in any split test. It helps to know how long the average customer takes to buy, and then NOT end the split test before that time - because there may be many latecomers.

That's one of the many mistakes people make when split testing, which I covered in Small Changes: Big Profits.

You need to make sure whatever split testing tool you use can still remember which version the visitor originally saw. My split testing tool Power Split Tester uses the standard cookie and IP address tracking, but there is also an option to track them via email if your autoresponder service allows you to add your own tracking code to each subscription. (This is an invisible field in the subscription form).

[tirtsa]

Those are all valid points raised, but I count them an integral part of the main weakness. One that simply cannot be overcome.
To get valid data, one needs to split test a significant number of visits over time. Most people do not test long enough to reach this number, as results are usually required within a couple of days.

Sure, you may have customers within your test period who's conversion was caused by a visit (buying decision) prior to the test period (same visitors are not counted more than once) and this is exactly why you need to test a large number of data (visitors). It's all about getting that valid and significant average.

I find the split testing tool offered by Logaholic Web Analytics (from logaholic.com) easy to use, providing me with the data needed.

[Daniel Scott]

Also keep in mind that in order to be reasonably "conclusive", your split test results must be significant - ie. one page must out-perform the other by about 15%.

I got this from Schwab and I have found it to be a good general guideline... though I'm not sure what the real testing junkies (like Paul) think.

-Dan

P.S. "Small Changes, Bit Profits" is a great read.