How Many Visitors needed to test New Copy?

I usually use 1,000 uniques as a good test sample.

Agreed, 1000 absolute minimum to get an idea on how it will convert. However I prefer 5000 to be absolutely sure on whether the copy is doing well, needs to be tweaked or scraped altogether.

Tyrus

Hey guys,

Yeah I'd concur with that.

1000 is going to be the bare
minimum for something like this.

Sales don't steadily come in at
1 per hundred visitors or 2 per
hundred or whatever, so you
need to get a wider sample
of stats to reflect that.

Hope that's useful

-David Raybould

You need to understand statistical significance

The general rule of thumb is based on the
number of actions taken: Number of sales,
optins, downloads, etc.

Assuming that you have X actions from your
control copy, and Y actions from your test
copy (where X is the better performing one).

If the difference in the number of actions
(X - Y) is greater than the square root of the
total number of actions (X + Y), then the
difference is statistically significant.

John

Hi.

I read an article by Andrew Reynolds recently and he was talking of a 5000 test!

1000 to 2000 should be good to tell you
as suppose you say your adword is costing you 20 cents .. ie $20 if 1000 ppl click
so you want to see how much are you making your profit over tht

Why are people guessing when the math, as John Taylor pointed out, is clear? This is math, not opinion.

Here is one place that answers your question. There are many like it:

Free Split Test Analyzer - Online Check For Statistical Significance

That's a handy little tool Bruce, thanks!

For those of you who don't understand that Math Lingo or stats quite so well...

Basically what these guys are saying is that you can't always just pick a number and say "hey, once it reaches XXX visitors, I know which page converts better!".

For example, let's say you have:

Page 1
Visits: 5000
Opt-ins: 1000

Page 2
Visits: 5000
Opt-ins: 1001

Even though you tested plenty of traffic, the difference was only 1 opt-in. Basically, it could just be luck that page 2 happened to get one more subscriber.

In this case, these pages either need significantly more testing or they may just be converting at about the same rate.

Hopefully that helps someone

Solomon Huey

P.S. In case you really wanted to hear a number, I like to test a minimum of 5,000 hits.

Now THIS is good stuff - thanks for the help and info.

I didn't know there was an online tool to calculate the "confidence value".

What if you had a page converting at 1 in 200 and you completely replaced the page with a new one and ran 1000 visitors to the page with ZERO conversions?

That little split test confidence meter gives you this:

Confidence = 100.00%
Statistical "z" value = -7.09

(assumes that you had 10,000 visitors to establish the 1 in 200 conversions)

BUT - if you just put in 100 visitors for version 2 - it still gives you 100% confidence that version 1 is the best page - in fact, if you just put in 1 visitor - it gives you the same answer.

So - while math is a good thing - that little online tool, to me anyway, is only good if you run an additional 10,000 visitors thru to page 2.

So the flaw is that whichever page has the most views (by a factor of 10 or more), that will be the page that the calculator says is the one that will perfom better.

If I enter 1 in 5000 and then 0 in 100 it gives me:

Version 1 5000 1 0.02%
Version 2 100 0 0.00%
Confidence = 68.27%
Statistical "z" value = -1.00
These results mean that there is a 68.27% chance that over time, Version 1 will continue to outperform Version 2. Or in other words, there's a 31.73% chance that the difference between their success rates was the result of random chance.


I think anyone can see that page 1 clearly stinks and the likelihood that page 2 will outperform it could be pretty good - but the analyzer says the opposite.