How long do I split test

13 replies
  • CRO
  • |
How long should I keep a split test running before the results become somewhat reliable?

100 clicks to each page? more? less?
#long #split
  • Profile picture of the author savidge4
    As I see it... the math behind A/B testing is Bells Theorem. Mathematically speaking the theorem falls into its "best" stride right at 1000 views per side of your test. All of the numbers just fall into that sweet spot where a definitive yes or no A is better than B answer can be given.

    That's my answer and Im sticking to it! LOL
    Signature
    Success is an ACT not an idea
    {{ DiscussionBoard.errors[9893971].message }}
    • Profile picture of the author BudiT
      Wow. Never heard of using Bell's theorem before. That's cool! Usually I just go 100 clicks for each side.
      Signature

      STOP wasting your money into hypes and BS.
      Turn your traffic and subscribers into White Hot Cash!

      Continue >>

      {{ DiscussionBoard.errors[9894482].message }}
      • Profile picture of the author savidge4
        Originally Posted by BudiT View Post

        Wow. Never heard of using Bell's theorem before. That's cool! Usually I just go 100 clicks for each side.
        Oh jeeze really? If you look at A/B Significance Test you can insert your numbers. and it is using bells theorem to figure out the overall significance of you test. If the numbers are to close, on the far right column last 2 lines it will tell you basically yes trust the results, or no, not enough data.

        Notice when you go to the page it is set for? 1000 LOL

        Here is another link that has many different calculators: http://www.evanmiller.org/ab-testing/
        including the top left one that helps you determine the sample size needed to run a decently accurate test.
        Signature
        Success is an ACT not an idea
        {{ DiscussionBoard.errors[9894585].message }}
        • Profile picture of the author Adrianhenry
          Originally Posted by savidge4 View Post

          Oh jeeze really? If you look at A/B Significance Test you can insert your numbers. and it is using bells theorem to figure out the overall significance of you test. If the numbers are to close, on the far right column last 2 lines it will tell you basically yes trust the results, or no, not enough data.

          Notice when you go to the page it is set for? 1000 LOL

          Here is another link that has many different calculators: Evan's Awesome A/B Tools - sample size calculator, A/B test results, and more
          including the top left one that helps you determine the sample size needed to run a decently accurate test.
          Thanks! Those are some great resources
          {{ DiscussionBoard.errors[9894615].message }}
          • Profile picture of the author richpage75
            Ideally, you will need at least 1000 unique visitors per week to the page that you want to test - otherwise it will take too long to get a test result (many weeks).

            And its not just about traffic too - you need to get enough 'conversions' (sales, leads, signups, subscribers etc) to get a valid test result. Ideally you will need at least 25 conversions per test variation (50 conversions per week in total).

            You should also take a full week to do the test - as this will help take into account variances in traffic and differences in traffic by day (weekend versus weekday etc), and this also allows enough time for fluctuations in test results to settle down.

            And if you don't have enough traffic to do testing, don't give up - there are still some good ways to test with lower traffic - I created a guide on A/B testing with low traffic to help with this.

            Hope that helps! Good luck with your testing...
            Signature
            Rich Page
            http://www.rich-page.com | Twitter: @richpage
            Author of 'Website Optimization: An Hour a Day'
            {{ DiscussionBoard.errors[9894925].message }}
  • Profile picture of the author CPABeyondNetwork
    Split test until you have enough data and then just scale the best campaign.

    Best regards,


    Affiliate Manager at CPABeyond
    Signature

    AM at CPA Beyond - #1 Private Network - Daily Payments (No Fees)

    {{ DiscussionBoard.errors[9897633].message }}
    • Profile picture of the author Adrianhenry
      Originally Posted by CPABeyondNetwork View Post

      Split test until you have enough data and then just scale the best campaign.

      Best regards,


      Affiliate Manager at CPABeyond
      I think this may be my favourite reply ever.

      How long do I split test for?

      "until you have enough data"

      ahhhh thanks, how did I miss that!
      {{ DiscussionBoard.errors[9899849].message }}
  • Profile picture of the author razorz18
    Test when you see clear winner. Its depend on your existing traffic. 100 visitors/clicks maybe not enough if you have 10k-20k per days. Just be practical on it.
    {{ DiscussionBoard.errors[9900649].message }}
    • Profile picture of the author savidge4
      Originally Posted by razorz18 View Post

      Test when you see clear winner. Its depend on your existing traffic. 100 visitors/clicks maybe not enough if you have 10k-20k per days. Just be practical on it.
      Im going to be THAT guy. there is nothing practical about A / B testing. it is what it is, a scientific method of comparison. I can not even begin to think how many times there was a "clear winner" early in testing and then once the full testing ran its course that "winner" ended up being the loser.

      So I keep pretty good notation on all of the tests I run. I just went and looked over the past 300 test that I have run. 68% of these tests had an early leader. out of those 204 tests with early leaders, 172 of those ended up being losers.

      that means to bail on the process early would have had me place the least converting option 84% of the time. Jeeze... that's bad! LOL
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[9901344].message }}
  • Profile picture of the author quinng123
    After I have at least 300 unique views daily, I do split testing for a month.

    This has generally worked out for me for my own experiences
    {{ DiscussionBoard.errors[9901743].message }}
    • Profile picture of the author savidge4
      Originally Posted by quinng123 View Post

      After I have at least 300 unique views daily, I do split testing for a month.

      This has generally worked out for me for my own experiences

      Right now you are split testing at 4500 impressions per side. Not a bad thing don't get me wrong. But if you mathematically only really need say 1000 per test variable, you could effectively be running 4.5 Different tests per month. Or one heck of a A B C D E F G H test LOL
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[9901940].message }}
  • Profile picture of the author lestatega
    I think that this is an endless game You always will compete and weed out the bad ones. You stop when you are in the range of 30% conversion rate.
    {{ DiscussionBoard.errors[9902598].message }}
  • Profile picture of the author FreedomBlogger
    The more the better! .. and as some recommended, at least 1000 clicks. Then track every single thing and look at the numbers to see what's working best!

    I hope this helps!

    Keep up the great work!

    I wish you the best!

    Cheers!
    Signature
    At the beginning, I thought making money online with a blog was super super hard. Not anymore. Learn the art of making money online blogging - step by step - HERE.
    {{ DiscussionBoard.errors[9902614].message }}

Trending Topics