Split Testing Scripts are not reliable

[Didier Faucher]

Do you think split testing scripts are reliable?

I doubt more and more...


I'm a test addict. I love testing. And I know how important it is.

I had my own custom made script developed 5 or 6 years ago,
and I use it all the time. I even have planed to released it for
free on the french speaking market, for which I registered the
domain name SplitTest.fr.

Problem is...

I'm confident of the quality of my script, and have relied on
it only until around 2 years ago.

Then I started to compare the number of sales (or subscribers)
recorded by my script and the number recorded by my
e-commerce application (or autoresponder).

Two years ago, 5-10% of sales were not recorded by my
split test script.

Last year, it was around 15-20%.

This year, it's around 25-30%.

I tried other scripts. And the results were the same: depending
the script and webpage, 20-30% of sales are not recorded by
the split test application!

These are enormous numbers.

This evening, I stumbled upon the following screenshot on
Chris Rempel's site:


(taken from http://www.thelazymarketer.com/vip-multiply/)

As you can see, 13 + 25 + 16 = 54 total sales, 16 were not
recorded by the split testing script. That's 29.6%!

So I'm not alone!


1- If this is "common" problem (Chris doesn't hide this fact on
the screenshot above, so I start to think I'm not alone and this
might be "normal" situation), and if 20-30% of actions are not
recorded by split test scripts, we can't be sure the results of
our tests are accurate.

2- Anyone else has noticed that?

3- Anybody knows why such enormous number of actions are
not recorded (cookies didn't work, but why)?

Sure, I'll hire a programmer... or 2... or 3 to try to fix this issue
on my SplitTest.fr script!

All the best.

Didier

[chris_surfrider]

Hi Didier,

Actually there was no problem on my end.

The reason why those sales weren't logged is because they didn't come from my target landing page.

In this case, my home page AND the split-tested landing URL were directly selling the product, and the home page produced 16 untracked sales in that case.

Interesting observation though.

Cookie-blocking would also affect this.

-Chris

[Didier Faucher]

Originally Posted by Jeff Hope

More people are blocking cookies entirely these
days, and more still are using anti-adware/spyware regularly, which very
often erases cookies.

Oh yes! This also explain why figures increase with time.

But 20-30%, that's a lot!

Maybe we can consider that these 20-30% untracked actions come from
*random* visitors, so tests are not biased...

Originally Posted by chris_surfrider

The reason why those sales weren't logged is because they didn't come
from my target landing page.

In this case, my home page AND the split-tested landing URL were directly
selling the product, and the home page produced 16 untracked sales in that
case.

Hi Chris!

How could you be sure the 16 untracked sales come from your homepage?

(I didn't notice your report is about split testing until now...
Obviously, I'm not trying to say your report is not good!)

Didier

[JonathanBoettcher]

Anything to do with javascript perhaps??? or is your script all in php? Just an idea because I know many people disable js entirely.

[Didier Faucher]

Originally Posted by JonathanBoettcher

Anything to do with javascript
perhaps??? or is your script all in php? Just an idea because I know
many people disable js entirely.

It's coded in PHP, and it doesn't use any JavaScript code.

"My" programmer doesn't have an explanation for now.

Didier

[Didier Faucher]

Sorry to bump this thread. This is not only about my own script.

Has anybody noticed 20-30% of untracked actions when using
a split testing script?

Thank you.

Didier

[radhika]

Two years ago, 5-10% of sales were not recorded by my
split test script.

Last year, it was around 15-20%.

This year, it's around 25-30%.

It might be depend on the method your script tracks the sale:

* Cookie method: As said above most people disable cookies / delete cookies often. So with out cookie, your script might not be able to track.

* IP number: If your script tracks the sale thru ip number from sales page to download (tracking page), the ip can be changed in between because the person bought your product different time from visiting your sales page OR using proxies.

* If every other thing is working, they bought visited the sales page at one time or using one computer and bought the product and goes to tracking page using different time and computer.

There are more variables that difficult to track with a sale.

.

[Didier Faucher]

Thank you for your input.

1) My script use both session and persistent cookies (set to one year).

It doesn't use IP numbers identification.

2) I tried to test using other scripts, and got the same results: around
25-30% of unregistered actions.

My main question and concern is: Is that the same with all split testing
scripts?

Can I safely say to the people who will use my script that they will
probably notice 20-30% of unregistered actions?

Didier

Originally Posted by radhika

It might be depend on the method your script tracks the sale:

* Cookie method: As said above most people disable cookies / delete cookies often. So with out cookie, your script might not be able to track.

* IP number: If your script tracks the sale thru ip number from sales page to download (tracking page), the ip can be changed in between because the person bought your product different time from visiting your sales page OR using proxies.

* If every other thing is working, they bought visited the sales page at one time or using one computer and bought the product and goes to tracking page using different time and computer.

There are more variables that difficult to track with a sale.

.

[chris_surfrider]

Didier,

I know those sales were from my homepage because the payment notification is tagged differently.

-CHris

[Chris Lockwood]

This isn't necessarily a big problem.

As long as the script ignores the untrackable sales and handles the other sales properly, you should still get valid results, right?

A big problem would be if the script did NOT ignore those untrackables and mistakenly assigned them to one of your sales letter versions.

In other words, if only 70% of sales can be properly tracked, the stats on which sales letter performed better should still be useful... certainly better than not testing at all.

[Paul Hancox]

Hi Didier

There's no FOOLPROOF way of tracking visitors using cookies and IP. That's because the cookie can be deleted or blocked, and IP addresses frequently change.

It's simply a FACT OF LIFE that not all sales will be tracked, even if you are using a combination of both cookie and IP address.

As Chris pointed out, as long as the split testing script doesn't just "guess" where the sale came from, you should be alright, as long as you're getting a big enough sample.

My program Power Split Tester can also track via a special code which can be incorporated into a mailing list, which is ideal if you have a sales process such as SQUEEZE PAGE > LIST > SALES LETTER.

You can incorporate the code into the list message, so the visitor will see the same version of the sales letter.

I created this feature because I kind of anticipated that cookies and IP tracking would gradually become more unreliable over time, as more people block them.

However, I don't think we'll get to the stage when the majority of users will have cookies blocked, because they rely on them for too many other things.

Hope this helps!

[Didier Faucher]

@ all: Thank you for your time and help!


@ Chris Rempel:

Ok. That's clear now.

Thank you.

Originally Posted by Chris Lockwood

As long as the script ignores the
untrackable sales and handles the other sales properly, you should
still get valid results, right?

Originally Posted by Paul Hancox

As Chris pointed out, as long as the split
testing script doesn't just "guess" where the sale came from, you should
be alright, as long as you're getting a big enough sample.

Not sure about that. And it's my real concern here.

It seems you say: just ignore the unrecorded sales.
I don't agree with that.


Simple example. 2 versions of a sales copy, A and B.

Total impressions is, say 1,234 (approximately 50% from A, 50% from B).

Sales from A: 48
Sales from B: 31

=> Winner is A (statistical confidence: 0.95)

Unrecorded sales (30% of the total): 33

That's not possible to know how much of these 33 sales were
presented A, and how much B.

Maybe 21 buyers saw A, and 12 B.
Or 10 saw A, and 23 saw B.
Or 1 saw A, and 32 B.

Meaning maybe the real result is B is the winner, or there's no
clear winner. One then runs the risk of coming to a wrong
conclusion.


So, in your opinion one has to make the hypothesis that in the
33 unrecorded sales, 50% were presented A, and 50% B (which,
in fact, consist of ignoring these 33 sales), otherwise that would
not be possible to draw a conclusion.

Is this hypothesis valid? I can't see a reason why it's valid, because
these 33 sales did happen. They're part of the sample.

Tester got 30% of sales from unknown origin.
It's a big bias that is introduced here if he doesn't take them into
account. IMO, we can't just ignore these sales.

Ignoring 10-15% "would be ok", but not 30%.


Also, from a pure mathematical point of view, if you remove 30%
of the sample, there's no way to be sure the statistical distribution
(in fact the "probability density function") is still gaussian ("bell curve"),
unless this part is taken at random.

So no conclusion can be drawn using the chi square test, for example
(PDF has to be gaussian for that).


That would be ok to remove 30% of the sample, if this "sub-sample"
was taken at random.

Problem is, these unrecorded sales are not taken at random...


You put 100 sales into a bag. You close your eyes and remove 30
from the total. That would be ok. But that's not what happens here.

Originally Posted by Paul Hancox

My program Power Split Tester can also track via a special code which can be incorporated into a mailing list, which is ideal if you have a sales process such as SQUEEZE PAGE > LIST > SALES LETTER.

You can incorporate the code into the list message, so the visitor will see the same version of the sales letter.

I created this feature because I kind of anticipated that cookies and IP tracking would gradually become more unreliable over time, as more people block them.

Yes, good! But how do you track returning visitors?



Didier

P.S.: Damn! I wish I could speak english better!

[Jeremy Wilson]

Didier, I see the same thing I've seen anywhere from 25-50% inaccuracies depending on the script being used.

One I've found to be very innacurate is Google's Adwords conversion tracking it is routinely off by about 30% no matter how I try to improve this I can't seem to get better results.

I still use it because I like to have all my stats right there in Adwords but it's frustrating to see it so far off.

[Paul Hancox]

Hi Didier

It seems you say: just ignore the unrecorded sales.
I don't agree with that.

I'm saying that, for scripts that rely solely on cookies and/or IP address, you HAVE to ignore them, because you cannot say where they have come from.

Is this hypothesis valid? I can't see a reason why it's valid, because
these 33 sales did happen. They're part of the sample.

Yes, but they should occur roughly in the same proportion as the sales that you could measure as knowing which version they originally saw.

In other words, if A and B converted at 60% and 40%, then over time, the unmeasured sales would also split at 60 / 40 as well.

The reason they're excluded, however, is you can't say for certain which version they originally saw. If you were to INCLUDE them at 50 / 50, you're much more likely to skew your stats, than if you EXCLUDED them.

Tester got 30% of sales from unknown origin.
It's a big bias that is introduced here if he doesn't take them into
account. IMO, we can't just ignore these sales.

Ignoring 10-15% "would be ok", but not 30%.

I understand what you're saying, but scripts using IP and cookies have no choice.

If you have the script "guess", then it's likely to skew the data much more than if it just says, "I don't know", and leaves the data out.

Yes, good! But how do you track returning visitors?

With my script, if you use the third tracking feature, you'd want as many of your returning visitors as possible to come via an email link - that way it can perfectly track which version they originally saw.

[Chris Lockwood]

Originally Posted by Didier Faucher

It seems you say: just ignore the unrecorded sales.
I don't agree with that.

So, in your opinion one has to make the hypothesis that in the
33 unrecorded sales, 50% were presented A, and 50% B (which,
in fact, consist of ignoring these 33 sales), otherwise that would
not be possible to draw a conclusion.

What is your solution then?

Nobody is saying that the unrecorded sales would be equally split between A and B. We're saying that we don't know where they came from, so we ignore them.

I can only think of 2 things you can do with untrackable sales: ignore them, or assign them to A or B using some guess.

I don't see how guessing gives better results, so the best option is to ignore them, given that they are untrackable.

Over a large enough sample size, the untrackable sales should have roughly the same split as the others - in other words if 60% of the recorded sales came from version A, then probably about 60% of the untrackable ones did, too, unless you have evidence to the contrary.

[Didier Faucher]

Originally Posted by Jeremy Wilson

Didier, I see the same thing I've seen
anywhere from 25-50% inaccuracies depending on the script being used.

Jeremy, thank you for your input. To be frank, that's not a good news
to know other scripts produce inaccurate results too.

I wish it's my script that is buggy.

Originally Posted by Chris Lockwood

I can only think of 2 things you can
do with untrackable sales: ignore them, or assign them to A or B using
some guess.

Third thing: you can't draw any conclusion, because there's a bias.

But... I got it...

Originally Posted by Chris Lockwood

Over a large enough sample size, the
untrackable sales should have roughly the same split as the others - in other
words if 60% of the recorded sales came from version A, then probably about
60% of the untrackable ones did, too, unless you have evidence to the contrary.

Originally Posted by Paul Hancox

if A and B converted at 60% and 40%, then
over time, the unmeasured sales would also split at 60 / 40 as well.

Chris and Paul, I understand what you say, and I didn't exclude this
possibility. I was just not sure that's a valid way to see things.

In fact, that's exactly what I was thinking when I started to see a large
percentage of unrecorded actions (around 2 years ago).

Until a couple of days... For some reason, I started to doubt. I have
planned to release my split testing script very soon, and was anxious
because of the large percentage of unrecorded actions. This explain
why I opened this thread, and why I opened it only few days ago.


I'm convinced (again) now.

Originally Posted by Paul Hancox

With my script, if you use the third tracking
feature, you'd want as many of your returning visitors as possible to come
via an email link - that way it can perfectly track which version they
originally saw.

Hmmm... Not sure I understand how your script tracks returning visitors.

How does it tag visitors to show them the correct version when they return?

Internet user clicks on the link, bookmark the page tested, then close her
browser, and stay offline for few days. She then goes online, and visit
the page tested again using the bookmark.

How your script "knows" which version this user saw the first time?

Do you use some secret reciepe?


// EDIT.

Paul, do you mean returning visitors have to click on the UTI link again?

//



Thank you everyone for your time and help!

Didier

[dangol]

In my case, in some heavy traffic sites, where I have a split test running I actually have to just give a share of the traffic to the test, because my script can't handle big traffic (about 40k uniques per day) because my server get's crazy (high load, memory, etc)

So what I do is to just send a part of my traffic to the test let say only 10k uniques instead of 40k, and I take that as a sample of the stats, when I get a winner I change it overall!

BTW what script are you using guys?

Thanks,
Dan