Split Testing Scripts are not reliable

17 replies
Do you think split testing scripts are reliable?

I doubt more and more...


I'm a test addict. I love testing. And I know how important it is.

I had my own custom made script developed 5 or 6 years ago,
and I use it all the time. I even have planed to released it for
free on the french speaking market, for which I registered the
domain name SplitTest.fr.

Problem is...

I'm confident of the quality of my script, and have relied on
it only until around 2 years ago.

Then I started to compare the number of sales (or subscribers)
recorded by my script and the number recorded by my
e-commerce application (or autoresponder).

Two years ago, 5-10% of sales were not recorded by my
split test script.

Last year, it was around 15-20%.

This year, it's around 25-30%.

I tried other scripts. And the results were the same: depending
the script and webpage, 20-30% of sales are not recorded by
the split test application!

These are enormous numbers.

This evening, I stumbled upon the following screenshot on
Chris Rempel's site:


(taken from http://www.thelazymarketer.com/vip-multiply/)

As you can see, 13 + 25 + 16 = 54 total sales, 16 were not
recorded by the split testing script. That's 29.6%!

So I'm not alone!


1- If this is "common" problem (Chris doesn't hide this fact on
the screenshot above, so I start to think I'm not alone and this
might be "normal" situation), and if 20-30% of actions are not
recorded by split test scripts, we can't be sure the results of
our tests are accurate.

2- Anyone else has noticed that?

3- Anybody knows why such enormous number of actions are
not recorded (cookies didn't work, but why)?

Sure, I'll hire a programmer... or 2... or 3 to try to fix this issue
on my SplitTest.fr script!

All the best.

Didier
#accuracy #reliable #scripts #split #split test #split testing #testing
  • Profile picture of the author chris_surfrider
    Hi Didier,

    Actually there was no problem on my end.

    The reason why those sales weren't logged is because they didn't come from my target landing page.

    In this case, my home page AND the split-tested landing URL were directly selling the product, and the home page produced 16 untracked sales in that case.

    Interesting observation though.

    Cookie-blocking would also affect this.

    -Chris
    Signature

    Making 6 Figures From Affiliate Marketing is Easier Than You Think. Here's Proof:

    http://www.TheLazyMarketer.com

    {{ DiscussionBoard.errors[212125].message }}
    • Profile picture of the author Didier Faucher
      Originally Posted by Jeff Hope View Post

      More people are blocking cookies entirely these
      days, and more still are using anti-adware/spyware regularly, which very
      often erases cookies.
      Oh yes! This also explain why figures increase with time.

      But 20-30%, that's a lot!

      Maybe we can consider that these 20-30% untracked actions come from
      *random* visitors, so tests are not biased...


      Originally Posted by chris_surfrider View Post

      The reason why those sales weren't logged is because they didn't come
      from my target landing page.

      In this case, my home page AND the split-tested landing URL were directly
      selling the product, and the home page produced 16 untracked sales in that
      case.
      Hi Chris!

      How could you be sure the 16 untracked sales come from your homepage?

      (I didn't notice your report is about split testing until now...
      Obviously, I'm not trying to say your report is not good!)

      Didier
      Signature

      Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

      {{ DiscussionBoard.errors[212175].message }}
  • Profile picture of the author JonathanBoettcher
    Anything to do with javascript perhaps??? or is your script all in php? Just an idea because I know many people disable js entirely.
    Signature
    {{ DiscussionBoard.errors[212201].message }}
    • Profile picture of the author Didier Faucher
      Originally Posted by JonathanBoettcher View Post

      Anything to do with javascript
      perhaps??? or is your script all in php? Just an idea because I know
      many people disable js entirely.
      It's coded in PHP, and it doesn't use any JavaScript code.

      "My" programmer doesn't have an explanation for now.

      Didier
      Signature

      Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

      {{ DiscussionBoard.errors[212214].message }}
  • Profile picture of the author Didier Faucher
    Sorry to bump this thread. This is not only about my own script.

    Has anybody noticed 20-30% of untracked actions when using
    a split testing script?

    Thank you.

    Didier
    Signature

    Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

    {{ DiscussionBoard.errors[214171].message }}
    • Profile picture of the author radhika
      Two years ago, 5-10% of sales were not recorded by my
      split test script.

      Last year, it was around 15-20%.

      This year, it's around 25-30%.
      It might be depend on the method your script tracks the sale:

      * Cookie method: As said above most people disable cookies / delete cookies often. So with out cookie, your script might not be able to track.

      * IP number: If your script tracks the sale thru ip number from sales page to download (tracking page), the ip can be changed in between because the person bought your product different time from visiting your sales page OR using proxies.

      * If every other thing is working, they bought visited the sales page at one time or using one computer and bought the product and goes to tracking page using different time and computer.

      There are more variables that difficult to track with a sale.

      .
      Signature
      Follow up Autoresponder PRO :: 33% Discount!!
      FREE Upgrades! IMPROVED Email Deliverability!!
      {{ DiscussionBoard.errors[214191].message }}
      • Profile picture of the author Didier Faucher
        Thank you for your input.

        1) My script use both session and persistent cookies (set to one year).

        It doesn't use IP numbers identification.

        2) I tried to test using other scripts, and got the same results: around
        25-30% of unregistered actions.

        My main question and concern is: Is that the same with all split testing
        scripts?

        Can I safely say to the people who will use my script that they will
        probably notice 20-30% of unregistered actions?

        Didier


        Originally Posted by radhika View Post

        It might be depend on the method your script tracks the sale:

        * Cookie method: As said above most people disable cookies / delete cookies often. So with out cookie, your script might not be able to track.

        * IP number: If your script tracks the sale thru ip number from sales page to download (tracking page), the ip can be changed in between because the person bought your product different time from visiting your sales page OR using proxies.

        * If every other thing is working, they bought visited the sales page at one time or using one computer and bought the product and goes to tracking page using different time and computer.

        There are more variables that difficult to track with a sale.

        .
        Signature

        Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

        {{ DiscussionBoard.errors[214239].message }}
        • Profile picture of the author chris_surfrider
          Didier,

          I know those sales were from my homepage because the payment notification is tagged differently.

          -CHris
          Signature

          Making 6 Figures From Affiliate Marketing is Easier Than You Think. Here's Proof:

          http://www.TheLazyMarketer.com

          {{ DiscussionBoard.errors[214260].message }}
  • Profile picture of the author Chris Lockwood
    This isn't necessarily a big problem.

    As long as the script ignores the untrackable sales and handles the other sales properly, you should still get valid results, right?

    A big problem would be if the script did NOT ignore those untrackables and mistakenly assigned them to one of your sales letter versions.

    In other words, if only 70% of sales can be properly tracked, the stats on which sales letter performed better should still be useful... certainly better than not testing at all.
    {{ DiscussionBoard.errors[214289].message }}
  • Profile picture of the author Paul Hancox
    Hi Didier

    There's no FOOLPROOF way of tracking visitors using cookies and IP. That's because the cookie can be deleted or blocked, and IP addresses frequently change.

    It's simply a FACT OF LIFE that not all sales will be tracked, even if you are using a combination of both cookie and IP address.

    As Chris pointed out, as long as the split testing script doesn't just "guess" where the sale came from, you should be alright, as long as you're getting a big enough sample.

    My program Power Split Tester can also track via a special code which can be incorporated into a mailing list, which is ideal if you have a sales process such as SQUEEZE PAGE > LIST > SALES LETTER.

    You can incorporate the code into the list message, so the visitor will see the same version of the sales letter.

    I created this feature because I kind of anticipated that cookies and IP tracking would gradually become more unreliable over time, as more people block them.

    However, I don't think we'll get to the stage when the majority of users will have cookies blocked, because they rely on them for too many other things.

    Hope this helps!
    Signature
    PresellContent.com - How to sell without "selling"
    {{ DiscussionBoard.errors[214517].message }}
    • Profile picture of the author Didier Faucher
      @ all: Thank you for your time and help!


      @ Chris Rempel:

      Ok. That's clear now.

      Thank you.



      Originally Posted by Chris Lockwood View Post

      As long as the script ignores the
      untrackable sales and handles the other sales properly, you should
      still get valid results, right?
      Originally Posted by Paul Hancox View Post

      As Chris pointed out, as long as the split
      testing script doesn't just "guess" where the sale came from, you should
      be alright, as long as you're getting a big enough sample.
      Not sure about that. And it's my real concern here.

      It seems you say: just ignore the unrecorded sales.
      I don't agree with that.


      Simple example. 2 versions of a sales copy, A and B.

      Total impressions is, say 1,234 (approximately 50% from A, 50% from B).

      Sales from A: 48
      Sales from B: 31

      => Winner is A (statistical confidence: 0.95)

      Unrecorded sales (30% of the total): 33

      That's not possible to know how much of these 33 sales were
      presented A, and how much B.

      Maybe 21 buyers saw A, and 12 B.
      Or 10 saw A, and 23 saw B.
      Or 1 saw A, and 32 B.

      Meaning maybe the real result is B is the winner, or there's no
      clear winner. One then runs the risk of coming to a wrong
      conclusion.


      So, in your opinion one has to make the hypothesis that in the
      33 unrecorded sales, 50% were presented A, and 50% B (which,
      in fact, consist of ignoring these 33 sales), otherwise that would
      not be possible to draw a conclusion.

      Is this hypothesis valid? I can't see a reason why it's valid, because
      these 33 sales did happen. They're part of the sample.

      Tester got 30% of sales from unknown origin.
      It's a big bias that is introduced here if he doesn't take them into
      account. IMO, we can't just ignore these sales.

      Ignoring 10-15% "would be ok", but not 30%.


      Also, from a pure mathematical point of view, if you remove 30%
      of the sample, there's no way to be sure the statistical distribution
      (in fact the "probability density function") is still gaussian ("bell curve"),
      unless this part is taken at random.

      So no conclusion can be drawn using the chi square test, for example
      (PDF has to be gaussian for that).


      That would be ok to remove 30% of the sample, if this "sub-sample"
      was taken at random.

      Problem is, these unrecorded sales are not taken at random...


      You put 100 sales into a bag. You close your eyes and remove 30
      from the total. That would be ok. But that's not what happens here.



      Originally Posted by Paul Hancox View Post

      My program Power Split Tester can also track via a special code which can be incorporated into a mailing list, which is ideal if you have a sales process such as SQUEEZE PAGE > LIST > SALES LETTER.

      You can incorporate the code into the list message, so the visitor will see the same version of the sales letter.

      I created this feature because I kind of anticipated that cookies and IP tracking would gradually become more unreliable over time, as more people block them.
      Yes, good! But how do you track returning visitors?



      Didier

      P.S.: Damn! I wish I could speak english better!
      Signature

      Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

      {{ DiscussionBoard.errors[215542].message }}
      • Profile picture of the author Paul Hancox
        Hi Didier

        It seems you say: just ignore the unrecorded sales.
        I don't agree with that.
        I'm saying that, for scripts that rely solely on cookies and/or IP address, you HAVE to ignore them, because you cannot say where they have come from.

        Is this hypothesis valid? I can't see a reason why it's valid, because
        these 33 sales did happen. They're part of the sample.
        Yes, but they should occur roughly in the same proportion as the sales that you could measure as knowing which version they originally saw.

        In other words, if A and B converted at 60% and 40%, then over time, the unmeasured sales would also split at 60 / 40 as well.

        The reason they're excluded, however, is you can't say for certain which version they originally saw. If you were to INCLUDE them at 50 / 50, you're much more likely to skew your stats, than if you EXCLUDED them.

        Tester got 30% of sales from unknown origin.
        It's a big bias that is introduced here if he doesn't take them into
        account. IMO, we can't just ignore these sales.

        Ignoring 10-15% "would be ok", but not 30%.
        I understand what you're saying, but scripts using IP and cookies have no choice.

        If you have the script "guess", then it's likely to skew the data much more than if it just says, "I don't know", and leaves the data out.

        Yes, good! But how do you track returning visitors?
        With my script, if you use the third tracking feature, you'd want as many of your returning visitors as possible to come via an email link - that way it can perfectly track which version they originally saw.
        Signature
        PresellContent.com - How to sell without "selling"
        {{ DiscussionBoard.errors[217239].message }}
      • Profile picture of the author Chris Lockwood
        Originally Posted by Didier Faucher View Post

        It seems you say: just ignore the unrecorded sales.
        I don't agree with that.

        So, in your opinion one has to make the hypothesis that in the
        33 unrecorded sales, 50% were presented A, and 50% B (which,
        in fact, consist of ignoring these 33 sales), otherwise that would
        not be possible to draw a conclusion.
        What is your solution then?

        Nobody is saying that the unrecorded sales would be equally split between A and B. We're saying that we don't know where they came from, so we ignore them.

        I can only think of 2 things you can do with untrackable sales: ignore them, or assign them to A or B using some guess.

        I don't see how guessing gives better results, so the best option is to ignore them, given that they are untrackable.

        Over a large enough sample size, the untrackable sales should have roughly the same split as the others - in other words if 60% of the recorded sales came from version A, then probably about 60% of the untrackable ones did, too, unless you have evidence to the contrary.
        {{ DiscussionBoard.errors[217249].message }}
        • Profile picture of the author Didier Faucher
          Originally Posted by Jeremy Wilson View Post

          Didier, I see the same thing I've seen
          anywhere from 25-50% inaccuracies depending on the script being used.
          Jeremy, thank you for your input. To be frank, that's not a good news
          to know other scripts produce inaccurate results too.

          I wish it's my script that is buggy.


          Originally Posted by Chris Lockwood View Post

          I can only think of 2 things you can
          do with untrackable sales: ignore them, or assign them to A or B using
          some guess.
          Third thing: you can't draw any conclusion, because there's a bias.

          But... I got it...


          Originally Posted by Chris Lockwood View Post

          Over a large enough sample size, the
          untrackable sales should have roughly the same split as the others - in other
          words if 60% of the recorded sales came from version A, then probably about
          60% of the untrackable ones did, too, unless you have evidence to the contrary.
          Originally Posted by Paul Hancox View Post

          if A and B converted at 60% and 40%, then
          over time, the unmeasured sales would also split at 60 / 40 as well.
          Chris and Paul, I understand what you say, and I didn't exclude this
          possibility. I was just not sure that's a valid way to see things.

          In fact, that's exactly what I was thinking when I started to see a large
          percentage of unrecorded actions (around 2 years ago).

          Until a couple of days... For some reason, I started to doubt. I have
          planned to release my split testing script very soon, and was anxious
          because of the large percentage of unrecorded actions. This explain
          why I opened this thread, and why I opened it only few days ago.


          I'm convinced (again) now.


          Originally Posted by Paul Hancox View Post

          With my script, if you use the third tracking
          feature, you'd want as many of your returning visitors as possible to come
          via an email link - that way it can perfectly track which version they
          originally saw.
          Hmmm... Not sure I understand how your script tracks returning visitors.

          How does it tag visitors to show them the correct version when they return?

          Internet user clicks on the link, bookmark the page tested, then close her
          browser, and stay offline for few days. She then goes online, and visit
          the page tested again using the bookmark.

          How your script "knows" which version this user saw the first time?

          Do you use some secret reciepe?


          // EDIT.

          Paul, do you mean returning visitors have to click on the UTI link again?

          //



          Thank you everyone for your time and help!

          Didier
          Signature

          Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

          {{ DiscussionBoard.errors[217591].message }}
          • Profile picture of the author dangol
            In my case, in some heavy traffic sites, where I have a split test running I actually have to just give a share of the traffic to the test, because my script can't handle big traffic (about 40k uniques per day) because my server get's crazy (high load, memory, etc)

            So what I do is to just send a part of my traffic to the test let say only 10k uniques instead of 40k, and I take that as a sample of the stats, when I get a winner I change it overall!

            BTW what script are you using guys?

            Thanks,
            Dan
            {{ DiscussionBoard.errors[217660].message }}
  • Profile picture of the author Jeremy Wilson
    Didier, I see the same thing I've seen anywhere from 25-50% inaccuracies depending on the script being used.

    One I've found to be very innacurate is Google's Adwords conversion tracking it is routinely off by about 30% no matter how I try to improve this I can't seem to get better results.

    I still use it because I like to have all my stats right there in Adwords but it's frustrating to see it so far off.
    {{ DiscussionBoard.errors[216592].message }}

Trending Topics