19 {{ upvoteCount | shortNum }}
19 {{ upvoteCount | shortNum }}

Split testing - what is minimum number to test with?

by tonyclif
15 replies
Hi
When doing your split testing, what is the minimum number of hits you analyse?
I'm thinking probably 50 of each option. That is, if I want to test performance of two pages, unless I get a huge difference immediately, I think I need to get 100 hits between the two options to see a pattern that I can rely on and decide which page to go with (and then test my next option against)
Hope this makes sense
Tony
#minimum #number #split
  • Profile picture of the author miklanderson2
    50 hits isn't usually enough when split testing unless you have a clear winner. If you have 50 hits and one page has 6 conversions and the other has none, you have a semi-clear winner. On the other hand, if one of the pages has 1 conversion and the other has 0, you're going to have to do more testing. I like to send at least a hundred hits and usually more to pages I'm split-testing.
    Signature

    "A bargain is something you don’t need at a price you can’t resist."
    -Franklin Jones

    {{ DiscussionBoard.errors[9355092].message }}
    • Profile picture of the author tonyclif
      Originally Posted by miklanderson2 View Post

      50 hits isn't usually enough when split testing unless you have a clear winner. If you have 50 hits and one page has 6 conversions and the other has none, you have a semi-clear winner. On the other hand, if one of the pages has 1 conversion and the other has 0, you're going to have to do more testing. I like to send at least a hundred hits and usually more to pages I'm split-testing.
      Thanks!
      I didn't consider the conversion rate compared to the hit rate, I was simply thinking about the hit rate.
      {{ DiscussionBoard.errors[9355115].message }}
  • Profile picture of the author AffGuides
    It really depends what you are advertising and how the split test performs. If you go through 100 clicks to 3 landing pages, and lp A gets a 20 conversions while LP b and c only get 5, that'd more than likely be statistically significant to me.

    If a gets 20, b gets 17 and c gets 25, that wouldn't be statistically significant, and you'd probably have to test 10 times that amount to get a clearer picture.

    Use this calculator to find out if you are statistically significant already or not. A/B Significance Test
    Signature
    Free CPA Affiliate Marketing Guide - Step-by-Step Hand Held Approach, Lander and Ad Templates, Real Case Studies! Get a Free Preview. Click Here!
    {{ DiscussionBoard.errors[9355099].message }}
    • Profile picture of the author tonyclif
      Originally Posted by AffGuides View Post

      It really depends what you are advertising and how the split test performs. If you go through 100 clicks to 3 landing pages, and lp A gets a 20 conversions while LP b and c only get 5, that'd more than likely be statistically significant to me.

      If a gets 20, b gets 17 and c gets 25, that wouldn't be statistically significant, and you'd probably have to test 10 times that amount to get a clearer picture.

      Use this calculator to find out if you are statistically significant already or not. A/B Significance Test
      Thanks for the tip. The A/b significance tester looks useful as well. It also makes you think differently about an increase in conversion of just a small number. I didn't think about the increase from say 6 to 9 conversions is a 50% uplift. I just saw it as "3" and a small number
      Tony
      {{ DiscussionBoard.errors[9355127].message }}
  • Profile picture of the author alvinhy
    Make sure when testing you are using the same traffic sources.
    Or else your results will be biased. towards that traffic source.

    Also split test one element at a time.
    For example test a different header, or colour.
    Signature
    >>> The King of Online Passive Income <<<
    www.JustPassive.com
    {{ DiscussionBoard.errors[9355244].message }}
  • Profile picture of the author NK
    I always try to get at least 100 clicks for testing conversion, but a bigger number like 500 would give you a more accurate representation of how well your page actually converts.
    Signature

    Affiliate Link Cloaking & Tracking Software - The most powerful tool available for affiliate marketers who are sick and tired of making way too little money!
    {{ DiscussionBoard.errors[9355249].message }}
    • Profile picture of the author tonyclif
      More great info alvinhy and NK. Thanks!

      There are so many things that we need to consider, and without the assistance of the WF contributors, I could make plenty of mistakes.

      Are there any good products, WSOs etc just on split testing?

      Tony
      {{ DiscussionBoard.errors[9355359].message }}
      • Profile picture of the author rayray13
        it really depends on your product that your trying to promote. if you dont mind me asking what program are you using for A/B split testing?
        {{ DiscussionBoard.errors[9355854].message }}
        • Profile picture of the author tonyclif
          Hi rayray
          Not using any program yet, but thinking about MyClickBoss - I really don't know much about this subject, other than it makes sense to test and measure everything we do, and I figured asking experienced people in here can only lead to info that will help me.
          Working well so far
          Tony
          {{ DiscussionBoard.errors[9357481].message }}
  • Profile picture of the author thisisraz65
    Hope that help you
    Split Test A/B Test Marketing Calculator - landing page-ppc-email split tests | SplitTestCalculator.com
    Signature
    No signature right now
    {{ DiscussionBoard.errors[9357921].message }}
    • Profile picture of the author criniit
      I always send at least 1,000 clicks to a split test before I ever even look at the results. Any less than that and it is not statistically significant.
      Signature
      {{ DiscussionBoard.errors[9358703].message }}
      • Profile picture of the author Alexa Smith
        Banned
        Originally Posted by criniit View Post

        I always send at least 1,000 clicks to a split test before I ever even look at the results. Any less than that and it is not statistically significant.
        So if you tested 800 clicks by sending 400 to each, and one group of 400 made 15 sales while the other didn't make a sale at all, that wouldn't be statistically significant for you? I don't mean it rudely, but I wonder whether you actually know what statistical significance means, if you honestly think you need a minimum of 1,000 clicks, regardless of context?

        It can't be determined from or satisfied by just the number of instances in the sample: it depends on the magnitude of the disparity between the outcomes, as well.

        Originally Posted by tonyclif View Post

        When doing your split testing, what is the minimum number of hits you analyse? I'm thinking probably 50 of each option.
        It depends what you're measuring, and how, Tony. For some things, a sample of 100 hits is actually going to be enough. For others, even 2,500 may not be enough to give you any reasonable degree of statistical significance. The answer's actually a little more complicated than some of the replies above suggest. It depends a lot on the variability/standard deviation in the observed results.

        If you explain a little more about what exactly you're measuring, it'll be much easier for people to help you.

        There's no point in people just plucking numbers out of the air and offering them, without knowing what you're testing and how, and what the variability might be. Guesses like that could easily be out by a factor of 10, or even 20 ... it just isn't possible to answer this question from the information you've provided so far, so beware of wild guesses which purport to do exactly that!

        .
        {{ DiscussionBoard.errors[9359000].message }}
  • Profile picture of the author thedanbrown
    If you're split testing a landing page, assuming it's a single variant test, I would test out 1000 clicks total, so 500 and 500. That should give you a pretty good gauge of what's performing better.
    Signature

    {{ DiscussionBoard.errors[9359193].message }}
  • Profile picture of the author Stuart Walker
    I like to run several thousand visitors to each before I jump to any conclusions.

    I've seen before with opt in boxes one being the clear winner after 1,000 people had viewed each then by 3,000 each the results being totally different and a different one being a clear winner.
    Signature
    ATTENTION WARRIOR: INCREDIBLE NEW POST AT NICHEHACKS.COM
    {{ DiscussionBoard.errors[9359539].message }}
    • Profile picture of the author tonyclif
      Alexa, great info to consider. I don't have anything set up yet to test on, but I am preparing an IM related Kindle book, and will have a free report accessed by a squeeze page, so it will be that squeeze page that I will be testing initially.

      Criniit, thanks for the input, though I definitely see what Alexa is saying in response to you and tend to agree
      {{ DiscussionBoard.errors[9360977].message }}

Trending Topics