Split Testing With Little To No Results..

by Dennis Miedema

Posted: 15 years ago 19 replies

INTERNET MARKETING

I'm at a point where I want to split test absolutely every single friggin' thing in my business..because guessing is taking RISKS, which equals losing money..while I want to test to have the answers I need to make more money.

But here's the thing: the rule of thumb is to split test two things simultaneously and see which one reaches 30 actions (whether thats clicks, opt ins or sales) first...BUT..how do you determine the winner of your split test when it will take you a really long time to reach that number or worse...when you CAN'T even reach that number?

I'm not a total newbie though, I know that there's a tool for seeing how certain you are about the winner (splittester.com), in which you enter the actions and the CTR values of both things you're testing..after which it gives you a % of certainty about who the winner is. So say you'll be 80% certain, then that's 20% guessing.

But what if you don't reach those numbers..those % or 30 clicks? I mean, it's a total paradox :confused: because..if you don't get 30 actions, you know you need to improve stuff..but if you never get those 30 actions then you don't know whether you're actually improving or making things WORSE :p

So say you're doing a split test with meager results, what do you recommend for determining the winner? Should I just wait until it reaches 30 in x amount of time (which could be weeks, or months..)?

I'm very curious how you guys would handle that..

#results #split

Chris Thompson

15 years ago

It's not as simple as seeing which group gets to 30 actions first. You need to understand more about stats, such as confidence intervals and stuff. OR you could just use an online calculator like this:
Split Test (A/B Test) Calculator

Thanks
1 reply

{{ DiscussionBoard.errors[906676].message }}

Dennis Miedema

15 years ago

Originally Posted by Chris Thompson

Great link, thanks Chris! Forgive me for asking, but where can I find out more about confidence intervals and other split test related things?

You see, I have NO idea which service I should use to split test things: hypertracker, 1shoppingcart, splittesting.com..there's so many of them..so do you or does anyone else know a great place to start split testing for beginners?

I want to A/B split test:
- my opt in page
- my sales letter
- my order form

And like I said..have no idea about where to start, and afterwards HOW: how can you let pages rotate without search engines going crazy, do you need some kind of tracking url, etc. So if anyone can get me started then well, I would seriously owe you a favor because get this..been in business for 6 months and although I'm already at break even, I haven't split tested a single thing yet!

Thanks

{{ DiscussionBoard.errors[906699].message }}

NewbiesDiary 15 years ago

you really should check out Conversion Chicken.

It's created by Tim Robinson who is a genius with this stuff! He's so good I've partnered with him for a few software products.

Conversion Chicken isn't my product and I don't receive any payment for it from Tim, it's just bloody brilliant software!!
- Thanks
{{ DiscussionBoard.errors[906718].message }}

Paul Hancox

15 years ago

Hi Dennis

I don't really think there's any statistically reliable way of determining a likely winner with "meagre" results, as you say. I demonstrated this in my book Small Changes: Big Profits, (which explains what you need to know about the stats, i.e. "confidence intervals", "significance" etc) where I ran a little experiment.

Basically, I flipped a coin 50 times, simulating a split test in which both versions A (heads) and B (tails) were exactly the same.

At first, A (heads) was ahead... and then B (tails) stormed ahead because of a string of tails, which can and often does happen. If we'd stopped the test earlier on, we'd have assumed B was better!

Only after about 30 results did A start to come back again. (Of course, the reality is that A and B were the same, but ending the test earlier would have told us something different!)

I'm skeptical of tools that can "predict" winners at such early stages... especially after only 5 or 10 actions. For example, in a toss of 10 coins, a 7/3 or 6/4 outcome is still quite likely - but this tells us NOTHING about which version is better.

Thanks
1 reply

Signature

PresellContent.com - How to sell without "selling"

{{ DiscussionBoard.errors[906796].message }}

Didier Faucher

15 years ago

Originally Posted by Paul Hancox

I'm skeptical of tools that can "predict" winners at such early stages... especially after only 5 or 10 actions. For example, in a toss of 10 coins, a 7/3 or 6/4 outcome is still quite likely - but this tells us NOTHING about which version is better.

- Yes, that's possible.

As usual, you'll get a result with a certain probability for being true, etc.

For small samples, one should use "Fisher's exact test", not chi-square test
(with or without Yates' correction) to test the null hypothesis.

(For larger samples, chi-square is not better, but p-value is easier to compute,
that's why it's used instead of Fisher's exact test).

- Statistics use probability theory, so no result is certain, never. Both for small
and large samples.

- There's a continuous debate among statisticians to know which method is
best, which one should or should not be used, in such and such conditions,
etc. So there's no easy answer to any statistics question.

- Statistics and probability are very very very complex theories.
Did I say complex?

- An article you may like:
The Insignificance of Statistical Significance Testing: What is Statistical
Hypothesis Testing?
NPWRC :: Statistical Significance Testing

Didier

Thanks

Signature

Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

{{ DiscussionBoard.errors[907174].message }}

TheAngelGuy 15 years ago

43splittests.com can give you some ideas so you don't have to reinvent the proverbial wheel.

But keep in mind, the one thing that people rarely test is the offer itself. Do that first, because that's your base number. Everything after that is a multiplier. You can have a great headline, but if your offer sucks, it doesn't matter.

Also, keep in mind that it's pretty useless to test down to the thousanth of a percentage point. You're probably splitting hairs over things where that same effort could be used toward creating another product.

Just some thoughts...
- Thanks
{{ DiscussionBoard.errors[906810].message }}
Dan Grossman 15 years ago

Pick a college, search its website for an introductory statistics course, and look at what textbook they use. Buy said text book. That's where you learn about confidence intervals and such. Statistics is required for just about every major outside the arts, including business of course.
- Thanks
Signature
Improvely: Built to track, test and optimize your marketing.
{{ DiscussionBoard.errors[906844].message }}
Dennis Miedema 15 years ago

Guys, thanks for all your help! I've decided to go with Hypertracker, but I'm having a hard time with applying the "programmy stuff". Why? Well, here's the deal...

- I'm now split testing the price of my product with a tracking url that goes like hypertracker.com slash something something..for the visitors that come to my site, so I can track sales and thus conversion, so I know which one is better

- I don't get a lot of visitors yet, but I got plenty of subscribers on my list, so here's the idea I had for faster results: use the same tracking url in me emails so I'll know faster! But there are two problems here..and I really, really hope that one of you can give me some pointers..

1. If I use the same tracking url, I can't track the conversion rate of my emails is that correct? Because the numbers would mix with the "normal" visitors..but since Im trying to see which page sells more, is that even a problem? And if I use a different tracking url, that means I constantly have to go to campaign 1 and 2 to add up the numbers so I know the sales, conversion, etc. and that doesn't make sense!

So how do I do that? Create a new campaign or not? Wow..I wish there would be a guide called "Split Testing Made Almost Too Easy" out there :S because this stuff is confusing me
- Thanks
{{ DiscussionBoard.errors[910854].message }}
Chris Thompson 15 years ago

Google Page Optimizer is another free service you can use to split test your web page elements. I am not sure how to use it with WordPress pages though, so I have not used it. I think it would be AMAZING if somebody came up with a WSO consisting of training videos to show me exactly how to use Google Page Optimizer on a wordpress blog to test my headline, copy, graphics, and other aspects of my page.

Price it under $20 and do a good job, I'm sold.
- Thanks
{{ DiscussionBoard.errors[910967].message }}
Dennis Miedema 15 years ago

Same here..there should be a product out there that covers all the basics of split testing without the techie mumbo jumbo..because Im not looking for philosophical discussions about statisticians, Im looking for here's where you need to go, here's what you need to do there, here's how to determine the winner, boom.

And leave the mumbo jumbo for the gold edition, lol

P.S. I already worked around that email problem
- Thanks
- 1 reply
{{ DiscussionBoard.errors[913375].message }}
- JohnMcCabe 15 years ago
  
  One thing I would add to Paul's post about his coin flipping trial...
  
  If you set up ten-flip trials, you can expect to get runs of ten heads, ten tails, and every variation in between. The probability of getting any one of those combinations is exactly the same, because you are dealing with a set of ten independent trials. In other words, just because you flipped a 'head' last time doesn't mean you're more likely to flip a 'tail' this time.
  
  Dennis, that's why it's difficult to tell you "here's where you need to go, here's what you need to do there, here's how to determine the winner, boom" using a small number of independent trials as your data set. There's just too much randomness.
  
  Even the so-called Law of Averages people like to quote for an A/B test like a coin flip is more properly called the Law of Large Numbers, which states that if you repeat the trial a very large number of times, the results will get closer and closer to the mean value.
  
  In other words, the longer you run an A/B test, the more closely your results will match the true pattern. It's the basic idea behind determining confidence levels. "How certain am I that the pattern in this set of trials actually represents the true pattern?"
  
  Thanks
  
  {{ DiscussionBoard.errors[913939].message }}
rtrotter 15 years ago

I have a CTR statistics calculator on my site that give you the confidence interval. Many of the online tools for this do not show the interval because they are really using the incorrect formula that in some cases result in a negative number for the lower limit. How can CTR be negative? My tool does the calculation correctly.

CTR Statistics Tool

Rodney
- [ 1 ] Thanks
Signature

Ping All Your Feed On Auto-Pilot
www.kping.com
{{ DiscussionBoard.errors[913962].message }}
Dennis Miedema 15 years ago

Thanks Rodney! Looks kinda cool! The tool actually says that the probablity statistical difference = 75,6%...does that mean Im 75% sure about the winner here? Please elaborate on how you should review the results of this calculator.

And oh...I've heard of taguchi split testing, well..by name that is..is it any good? I know you created it and everything, but lets just say I'm not the worlds greatest student of numbers which means I need a clear and thorough explanation of the technical stuff..
- Thanks
{{ DiscussionBoard.errors[917510].message }}
rtrotter 15 years ago

Hi Dennis, the 75% probability of statistical difference means you can be 75% sure if you pick to winning version, it will continue to out perform the other version. I really like to see a 95% probability.

I tried to design the software in a way that allows any non-technical person to use it. You do not need to understand the Taguchi method and it tells you when the test is significant. It is all that technical stuff that keeps people from testing and I wanted to do something better.

Rondey
- Thanks
Signature

Ping All Your Feed On Auto-Pilot
www.kping.com
{{ DiscussionBoard.errors[927309].message }}
Didier Faucher 15 years ago

Rodney ("rtrotter"), what do you think the future of Taguchi tests will be?

I created my own Taguchi tests application (target is the french-speaking
market), but not sure I'll sell it, because:

1. Tests and results are very hard to understand for non-technical people,

2. Test are quite hard and long to design (it makes no sense to test random
variables and options).

Didier
- Thanks
Signature

Saya Mau Makan Angin - I Want To eat The Wind (Why not?)
{{ DiscussionBoard.errors[927672].message }}

SteveJohnson

15 years ago

I swear I just saw this amorphous cloud, embedded with whispery remnants and snippets of statistical terms and theories, float ever so gently above my head, just out of reach. Then I turned away in disappointment and it zapped my ass with a bolt of lightning. Alas, whatever bits of knowledge the lightning contained were of no use.

Y'alls was speaking Greek...

Thanks
1 reply

Signature

The 2nd Amendment, 1789 - The Original Homeland Security.

Gun control means never having to say, "I missed you."

{{ DiscussionBoard.errors[927703].message }}

Didier Faucher

15 years ago

Originally Posted by SteveJohnson

I swear I just saw this amorphous cloud, embedded with whispery remnants and snippets of statistical terms and theories, (...)

Y'alls was speaking Greek...

If my English was better, I would try to explain technical words
in layman's terms.

Sorry for not being of help.

Didier

Thanks

Signature

Saya Mau Makan Angin - I Want To eat The Wind (Why not?)

{{ DiscussionBoard.errors[927745].message }}

SteveJohnson 15 years ago

If my English was better, I would try to explain technical words
in layman's terms.

Sorry for not being of help.

I was being a little facetious. I understand the terms. I have a faint grasp of the theories (VERY faint ).

At some point in the future, this might hold me back from astounding success, but I just don't see the value in applying this to IM to the degree that you are discussing (for the average or slightly above-average IMer). I could easily get so immersed in numbers that I forget the one true measure that counts - are people buying?
- Thanks
Signature

The 2nd Amendment, 1789 - The Original Homeland Security.

Gun control means never having to say, "I missed you."
{{ DiscussionBoard.errors[927818].message }}
rtrotter 15 years ago

Steve, "are people buying" is the target of testing and founding out how to get MORE people buying is the goal. What would happen to your income if on the same amount of traffic you are getting today, you had a 25% increase in buyers?

Didier, the future of Taguchi (or any other approach) is in how easy it is to use. If people with no background in statistics or testing can use it and see improvement in their conversion of visitor to buyer then it is a good solution.

The problem with all of the Taguchi applications is the user has to choose the array to use. Are they going to know how to do that? NO. So the solution is to write the application so it is done for them and they do not have to think about it. You also write the program so that it tells the user when the test is significant and gives them one button to click that say, "Use This Design".

The point I am making is testing software will be used when it is a no brainer. I set out to write software that you could train a monkey to use and I think I've come pretty close.

Rodney
- [ 1 ] Thanks
Signature

Ping All Your Feed On Auto-Pilot
www.kping.com
{{ DiscussionBoard.errors[928725].message }}

Split Testing With Little To No Results..

Trending Topics

Been a while! Any of you Old Timers here?

How much does the rebranding affect CR ?

Are billboards still effective in driving customers?

Boat and Jet Ski Rentals...... Door Hangers Effective?

Clickbank low cost recurring