how many clicks do I need for each ad unit for split testing to know which one to stop?

skyla 13 years ago

I was also told to wait for 1,000 clicks before deciding.
- Thanks
Signature
Buy PLR Today
Practical PLR FB Group
Ways to Earn Money Writing
{{ DiscussionBoard.errors[3606631].message }}

Alexa Smith

Banned 13 years ago

Originally Posted by kayo1234

14 days no matter how many clicks

Originally Posted by Anonymous Affiliate

I also decide after 1,000 clicks.

Respectfully, these answers are both guesswork.

A more accurate approach is to monitor until a degree of statistical significance is observed in the results, according to the standard deviation ("variability"); in this way, it can be said with accuracy that the probability of "site-A" (for example) genuinely being the better/best one is 90%/95% or whatever parameter you've decided is acceptable to you as "proof".

It's done in the same way as "publishable figures from a clinical trial" to prove that treatment A is better than treatment B, typically in those cases with a "p of < 0.05" (in other words there's more than a 95% chance that the results observed, with reference to the variability, are not caused by chance alone).

With respect, to have a fixed time-period without reference to the volume of traffic, and/or to have a fixed number of clicks without reference to the standard deviation between the results, are both just "guessing".

Thanks
1 reply

{{ DiscussionBoard.errors[3606754].message }}

E. Brian Rose

13 years ago

Originally Posted by Alexa Smith

Respectfully, these answers are both guesswork.

A more accurate approach is to monitor until a degree of statistical significance is observed in the results, according to the standard deviation ("variability"); in this way, it can be said with accuracy that the probability of "site-A" (for example) genuinely being the better/best one is 90%/95% or whatever parameter you've decided is acceptable to you as "proof".

It's done in the same way as "publishable figures from a clinical trial" to prove that treatment A is better than treatment B, typically in those cases with a "p of < 0.05" (in other words there's more than a 95% chance that the results observed, with reference to the variability, are not caused by chance alone).

With respect, to have a fixed time-period without reference to the volume of traffic, and/or to have a fixed number of clicks without reference to the standard deviation between the results, are both just "guessing".

Or just gather 1000 clicks from each of your tests and conclude.

Thanks
1 reply

Signature

Founder of JVZoo. All around good guy :)

{{ DiscussionBoard.errors[3609676].message }}

DudleyDog

13 years ago

1000 clicks can be a long time and costly too.

You can reduce this dramatically by using a statistical significance test based on 'Pearsons Chi-Square'. I'll try to simplify as follows:

I always do A/B testing of ads, so that's two at a time folks.

Example

If Ad1 has 44 clicks and Ad2 has 21 clicks over the same time (and that's important). Which one is performing better?

You need do a couple of calculations:

Let N equal the number of trials. In this case N = 44 + 21 = 65

Let D equal half of the difference between Ad1 an Ad2. So D = 44 - 21 divided by 2 = 11.5

Calculate D squared = 11.5 x 11.5 = 132.25

The test is significantly different if D squared is bigger than N.

In this case D squared is 132.25 and N is 65. Therefore you can say with confidence that Ad1 is performing better than Ad2.

Now try it with different numbers:

Ad1 has 44 clicks and Ad2 has 35

N = 44 + 35 = 79
D = 44 - 35 divided by 2 = 4.5
D squared = 4.5 x 4.5 = 20.25
This time D squared is less than N so the test is inconclusive. You have to let it run until D squared is greater than N. Then you can make a decision.

[ 4 ] Thanks
2 replies

Signature

Find Out What Really Works In Web Design
Mobile Website Design | Website Design | QR Code Marketing

{{ DiscussionBoard.errors[3609999].message }}

52.ct

12 years ago

Originally Posted by DudleyDog

1000 clicks can be a long time and costly too.

You can reduce this dramatically by using a statistical significance test based on 'Pearsons Chi-Square'. I'll try to simplify as follows:

I always do A/B testing of ads, so that's two at a time folks.

Example

If Ad1 has 44 clicks and Ad2 has 21 clicks over the same time (and that's important). Which one is performing better?

You need do a couple of calculations:

Let N equal the number of trials. In this case N = 44 + 21 = 65

Let D equal half of the difference between Ad1 an Ad2. So D = 44 - 21 divided by 2 = 11.5

Calculate D squared = 11.5 x 11.5 = 132.25

The test is significantly different if D squared is bigger than N.

In this case D squared is 132.25 and N is 65. Therefore you can say with confidence that Ad1 is performing better than Ad2.

Now try it with different numbers:

Ad1 has 44 clicks and Ad2 has 35

N = 44 + 35 = 79
D = 44 - 35 divided by 2 = 4.5
D squared = 4.5 x 4.5 = 20.25
This time D squared is less than N so the test is inconclusive. You have to let it run until D squared is greater than N. Then you can make a decision.

Thanks for your explanation. Quick question though. Would it not be better to use conversion rate as opposed to the number of clicks? In the first example, just because Ad1 one got 44 clicks and Ad2 got 21 does not necessarily mean Ad1 converted more than Ad2.
In fact, it is possible that Ad2 converted more with less clicks.

Thanks
1 reply

Signature

Rocket Spanish review
Rocket Spanish review blog

{{ DiscussionBoard.errors[5917281].message }}

DudleyDog

12 years ago

Originally Posted by 52.ct

Thanks for your explanation. Quick question though. Would it not be better to use conversion rate as opposed to the number of clicks? In the first example, just because Ad1 one got 44 clicks and Ad2 got 21 does not necessarily mean Ad1 converted more than Ad2.
In fact, it is possible that Ad2 converted more with less clicks.

This is meant to measure the effectiveness of your competing ads. The objective of your ad is to get someone to click it and take them to your site. To try and convert at the ad is too early in the conversion process.

The combination of your ad content and what they find on your landing page will affect conversion rate. What I mean is there should be some continuity between what is in your ad and what is on your landing page.

You need to test ad against ad and landing page against landing page.

Thanks

Signature

Find Out What Really Works In Web Design
Mobile Website Design | Website Design | QR Code Marketing

{{ DiscussionBoard.errors[5917832].message }}

Lucian Lada

11 years ago

Originally Posted by DudleyDog

1000 clicks can be a long time and costly too.

You can reduce this dramatically by using a statistical significance test based on 'Pearsons Chi-Square'. I'll try to simplify as follows:

I always do A/B testing of ads, so that's two at a time folks.

Example

If Ad1 has 44 clicks and Ad2 has 21 clicks over the same time (and that's important). Which one is performing better?

You need do a couple of calculations:

Let N equal the number of trials. In this case N = 44 + 21 = 65

Let D equal half of the difference between Ad1 an Ad2. So D = 44 - 21 divided by 2 = 11.5

Calculate D squared = 11.5 x 11.5 = 132.25

The test is significantly different if D squared is bigger than N.

In this case D squared is 132.25 and N is 65. Therefore you can say with confidence that Ad1 is performing better than Ad2.

Now try it with different numbers:

Ad1 has 44 clicks and Ad2 has 35

N = 44 + 35 = 79
D = 44 - 35 divided by 2 = 4.5
D squared = 4.5 x 4.5 = 20.25
This time D squared is less than N so the test is inconclusive. You have to let it run until D squared is greater than N. Then you can make a decision.

Is there a way to adapt this formula to an A/B/C/D split-test?
Also, Paul Hancox in his ebook, Big Profits, has this other formula:

You find the square root of the total number of actions (So it would be square root of A + B), then find if the number resulted is smaller than the difference between A and B, so basically this is the formula:

Code:

IF [square root of (A+B)] < (A-B), then it's statistically significant.

For people who like math... is there a difference between the two formulas? And can one be adopted to a 4-variation test?

Thanks
1 reply

Signature

Professional autoresponder email writer.
I also write premium quality articles, e-books, etc.

{{ DiscussionBoard.errors[7816847].message }}

butters

11 years ago

Originally Posted by Lucian Lada

Is there a way to adapt this formula to an A/B/C/D split-test?
Also, Paul Hancox in his ebook, Big Profits, has this other formula:

You find the square root of the total number of actions (So it would be square root of A + B), then find if the number resulted is smaller than the difference between A and B, so basically this is the formula:

Code:

IF [square root of (A+B)] < (A-B), then it's statistically significant.

For people who like math... is there a difference between the two formulas? And can one be adopted to a 4-variation test?

This may help, took it from my Measurements unit at uni... Just follow the chart and it will tell you what statistical test you need to preform

If you need help understand parametric and non parametric criteria then let me know.

[ 1 ] Thanks

{{ DiscussionBoard.errors[7816911].message }}

Valtan

13 years ago

1000 click it is mate

Thanks

{{ DiscussionBoard.errors[3606636].message }}

powerspike

13 years ago

As stated above, at least 1,000 clicks.

The reason you want quite a few clicks, is sometimes you might have the first 4 people buy an item (or take the required action), and that'd skew your data. Generally around 1000 is a good number to take that your stats are on point.

Obviously the more you willing to do, the better your testing will be =)

Thanks

{{ DiscussionBoard.errors[3606647].message }}

Saito

12 years ago

Who on earth has the money to get 1000 clicks to test one ad? $1000-3000 or more for one ad? $90,000 to test 30 ads at $3 per click? Seriously? I do not believe that's necessary.

Perry Marshall says you can get away with 30 clicks per ad to get a rough feel. Another article I found by some serious Search Engine marketers says 100 is good, maybe 50 but they don't like to.

So I believe that rough rules of thumb DO exist and if anyone can add to or challenge these smaller numbers, I would appreciate it, as I have found that few things in life are really as vague as "just test."

There are also sites like SplitTester.com and Vertster.com/adwords-tool/ that do the calculations for you and can tell you there's an 85% probability that one ad is better than the other. I'd rather be 80% sure than spend hundreds more to be 100% sure.

Thanks

Signature

Do You Suffer From Incomplete Funnel Syndrome?

We Unlock Your Hidden Profit Centers With Quality Info Products & Digital Content

{{ DiscussionBoard.errors[5662122].message }}

Saito

12 years ago

What are your thoughts on the # of impressions when using the content network?

Any good rules of thumb for ads' CTR there, too? I got .60% on one and was very pleased but not sure what the norm is, as I believe many have said 1.00% or higher is usually acceptable in the Search Network for an ad's CTR.

Thanks

Signature

Do You Suffer From Incomplete Funnel Syndrome?

We Unlock Your Hidden Profit Centers With Quality Info Products & Digital Content

{{ DiscussionBoard.errors[5689882].message }}

TheWonderingWanderer

12 years ago

I usually go for a bit more than 1000 clicks but it is a good starting point.

Thanks
1 reply

{{ DiscussionBoard.errors[5917310].message }}

Chris Thompson 12 years ago

Anyone who gives you a concrete answer to this question is WRONG.

You can't say "wait for 1000 clicks" or "wait 14 days". Saying this demonstrates a lack of understanding.

Google "split test calculator" and you'll come up with a variety of actual CORRECT ways to determine if you have a winner or not. It's based on statistics, and what matters is not only the overall sample size, but the difference in conversion (or click through rate, in the case of ads) between your original and your variation.

See the LAST section of this split testing tutorial near the bottom:
http://blog.outsourcefactor.com/spli...everyone-else/
- Thanks
{{ DiscussionBoard.errors[5917605].message }}

thinktwice

12 years ago

No more than 2 ads at a time.

Wait 200-300 clicks... then remove the worse one. Add a new one, restart.

Doing 4 at the same time will not give good results. You really need to split in 2.

Thanks
1 reply

{{ DiscussionBoard.errors[5918189].message }}

Chris Thompson

12 years ago

Originally Posted by thinktwice

No more than 2 ads at a time.

Wait 200-300 clicks... then remove the worse one. Add a new one, restart.

Doing 4 at the same time will not give good results. You really need to split in 2.

This is again absolutely the WRONG way to go. You need to calculate the winner. Stop guessing.

And there's nothing wrong with serving up 3 or 4 ads at a time. This is done all the time with split testing. It's highly effective, and you'll find plenty of case study examples on the visual website optimizer blog. Testing ads is no different.

Thanks
1 reply

{{ DiscussionBoard.errors[5918404].message }}

DudleyDog 12 years ago

Plus you can easily not have winner afte x amount of clicks.
- Thanks
- 1 reply
Signature

Find Out What Really Works In Web Design
Mobile Website Design | Website Design | QR Code Marketing
{{ DiscussionBoard.errors[5920293].message }}
- Chris Thompson 12 years ago
  
  Originally Posted by DudleyDog
  
  Plus you can easily not have winner afte x amount of clicks.
  
  Exactly! In fact you never really know for certain that you have a winner until your calculation shows 100% certainty, but the good split testing tools like GWO, or Visual Website Optimizer show you the % chance of your variation beating your control. So you pick the % you're happy with achieving.
  
  Same calc applies to ads.
  
  Thanks
  
  1 reply
  
  {{ DiscussionBoard.errors[5921180].message }}
  
  nolite 12 years ago
  
  Are there any good tools, or Saas platforms for automatically rotating adds and giving statistical significance, like VWO?
  
  Thanks
  
  {{ DiscussionBoard.errors[5998729].message }}

Lucian Lada

11 years ago

Lee, thanks for your help, but I don't understand a word of it. I need a For Dummies version of it, and I know that you don't have one. Still, thanks.

Thanks
1 reply

Signature

Professional autoresponder email writer.
I also write premium quality articles, e-books, etc.

{{ DiscussionBoard.errors[7816970].message }}

butters 11 years ago

Originally Posted by Lucian Lada

Lee, thanks for your help, but I don't understand a word of it. I need a For Dummies version of it, and I know that you don't have one. Still, thanks.

I can reference you a scientific book which is all about using statistical tests (In science, needs to be contextually adapted for IM if you want .) Also, there is software out there called SPSS, it is a software which automatically does these tests for you, you just have to plug in the data. The only problem is, you need to understand the test you are doing before you actually do the test or you wont understand what you see .
- Thanks
{{ DiscussionBoard.errors[7816992].message }}