Significant Split Test Results?

TimS · 11-06-2009, 07:35 PM

I have a question for those of you that regularly run split tests.

I was hoping for something that would jump out and slap me in the face, but the results seem almost insignificant.

After about 200 subscribers, running an A/B split test, one page resulted in 94 subscribers, the other is at 104.

Is that enough to consider it a significant run and start testing the better page against another?

Thanks

jaggyjay · 11-06-2009, 10:16 PM

No - those numbers aren't enough to be statistically significant. Are the pages very similar? If feasible, you'll want to continue testing until you get statistically significant results. If pressed for time, just make a decision and go with the better producing one.

MikeHumphreys · 11-06-2009, 10:20 PM

Quote:

Originally Posted by jaggyjay

No - those numbers aren't enough to be statistically significant. Are the pages very similar? If feasible, you'll want to continue testing until you get statistically significant results. If pressed for time, just make a decision and go with the better producing one.

I agree with Jaggyjay... definitely not statistically significant.

The more important question is, why isn't the testing software you are using telling you if it's statistically significant or not? That's a pretty common feature in many commercial testing scripts.

Frank Bruno · 11-06-2009, 10:29 PM

I would use the better version and use it as the "control" and split test another version against the current winning version.

Frank Bruno

mequad · 11-06-2009, 10:39 PM

Yes JaggyJay...

TimS should use much different copy angles to test,
not just something slightly different.

Add video, or short vs. long copy... not just changing a word
here or there.

Although simple headline changes can often make a huge difference.

I think the approximate 100 ACTIONS from each test IS significant
enough to pick the winner, although it is only about 10% better in
this case.

If it is 100 exposures (not actions), then no...

But 100 ACTIONS/CONVERSIONS is significant, whether it is
opt-ins, or sales.

Some statisticians say 50 ACTIONS/CONVERSIONS by both controls
is enough to be 90% certain of the winner.

From my learning and experience...

mequad · 11-06-2009, 10:44 PM

Hello Frank... from one of your happy clients.

TimS · 11-07-2009, 05:30 AM

Thanks for the replies. I'm not using an A/B script, I'm using a PHP script that pulls one of the two pages at random. It loads them evenly over the course of time and gives me a good split.

The pages are very different. One is with a headline and copy, the other is with a link that asks them to fill out the form w/ a video included on the page. The text is well below the fold.

Thanks,
Tim

11-06-2009, 07:35 PM	#1
TimS ArticleToolChest.com War Room Member Join Date: Apr 2005 Location: Boynton Beach, Florida Posts: 538 Thanks: 3 Thanked 75 Times in 13 Posts Social Networking	Significant Split Test Results? I have a question for those of you that regularly run split tests. I was hoping for something that would jump out and slap me in the face, but the results seem almost insignificant. After about 200 subscribers, running an A/B split test, one page resulted in 94 subscribers, the other is at 104. Is that enough to consider it a significant run and start testing the better page against another? Thanks
	Free Article Writing Software - Write, Rewrite, Spin and Blend your Online Content

11-06-2009, 10:16 PM	#2
jaggyjay Enlightened Warrior War Room Member Join Date: Nov 2007 Location: New York, USA. Posts: 337 Thanks: 3 Thanked 27 Times in 27 Posts	Re: Significant Split Test Results? No - those numbers aren't enough to be statistically significant. Are the pages very similar? If feasible, you'll want to continue testing until you get statistically significant results. If pressed for time, just make a decision and go with the better producing one.
	Please be sure to hit the "Thanks" button if this post has helped you.

11-06-2009, 10:29 PM	#4
Frank Bruno Senior Warrior Member War Room Member Join Date: Jun 2003 Location: USA Posts: 2,341 Blog Entries: 1 Thanks: 7 Thanked 73 Times in 58 Posts Social Networking Contact Info	Re: Significant Split Test Results? I would use the better version and use it as the "control" and split test another version against the current winning version. Frank Bruno
	- WSO- New "Systemizer" Mind Mapping Software For Internet Marketers Click Here! - Secret Society Coaching- Limited Members- Get Yourself In Now! Click Here! - Video Marketers Club "The Holy Grail Of Video Marketing" Click Here

11-06-2009, 10:39 PM	#5
mequad Dr. Michael Quadlander War Room Member Join Date: Aug 2007 Location: Texas...And the place I call home, Is just south of Austin, in the Hill Country north of San Antone! Posts: 28 Thanks: 8 Thanked 3 Times in 3 Posts	Re: Significant Split Test Results? Yes JaggyJay... TimS should use much different copy angles to test, not just something slightly different. Add video, or short vs. long copy... not just changing a word here or there. Although simple headline changes can often make a huge difference. I think the approximate 100 ACTIONS from each test IS significant enough to pick the winner, although it is only about 10% better in this case. If it is 100 exposures (not actions), then no... But 100 ACTIONS/CONVERSIONS is significant, whether it is opt-ins, or sales. Some statisticians say 50 ACTIONS/CONVERSIONS by both controls is enough to be 90% certain of the winner. From my learning and experience...
	Creating Products That ACCELERATE YOUR SUCCESS = Click Here \| Mind-Mapping For Success = Click Here *Dr. Michael Quadlander*

11-06-2009, 10:44 PM	#6
mequad Dr. Michael Quadlander War Room Member Join Date: Aug 2007 Location: Texas...And the place I call home, Is just south of Austin, in the Hill Country north of San Antone! Posts: 28 Thanks: 8 Thanked 3 Times in 3 Posts	Re: Significant Split Test Results? Hello Frank... from one of your happy clients.
	Creating Products That ACCELERATE YOUR SUCCESS = Click Here \| Mind-Mapping For Success = Click Here *Dr. Michael Quadlander*

11-07-2009, 05:30 AM	#7
TimS ArticleToolChest.com War Room Member Join Date: Apr 2005 Location: Boynton Beach, Florida Posts: 538 Thanks: 3 Thanked 75 Times in 13 Posts Social Networking	Re: Significant Split Test Results? Thanks for the replies. I'm not using an A/B script, I'm using a PHP script that pulls one of the two pages at random. It loads them evenly over the course of time and gives me a good split. The pages are very different. One is with a headline and copy, the other is with a link that asks them to fill out the form w/ a video included on the page. The text is well below the fold. Thanks, Tim
	Free Article Writing Software - Write, Rewrite, Spin and Blend your Online Content