How not to run an A/B test

mattbarrie · 31st May 2014, 12:03 AM

A great writeup by Evan Miller on the principles of A/B testing:

How Not To Run An A/B Test

richpage75 · 12th Jun 2014, 03:03 AM

To me, this type of article mentioning stats formulas scares off too many people from doing A/B testing. Much more important is how you generate test ideas - always use a test hypothesis, and always create insights from your web analytics tools and visitor feedback.

Doing that will ensure you get better test results that get test significance much quicker, rather than getting bogged down by understanding statistics and potential 'analysis paralysis'.

And don't forget, doing A/B testing is just a small piece of the CRO puzzle! And many sites don't even have enough traffic to do A/B testing - but they can still make improvements based on CRO best practices and do better analysis of their website and visitors.

jacobmark · 8th Aug 2014, 12:44 PM

Google has a simple a/b split testing platform. You can test multiple different pages, you just need to pump a lot of traffics to the test pages.

31st May 2014, 12:03 AM	#1
mattbarrie Freelancer.com CEO Join Date: 2010 Posts: 127 Thanks: 36 Thanked 411 Times in 49 Posts	How not to run an A/B test Share on: A great writeup by Evan Miller on the principles of A/B testing: How Not To Run An A/B Test

12th Jun 2014, 03:03 AM	#2
richpage75 Conversion Rate Optimizer Join Date: 2013 Location: Cologne Posts: 29 Thanks: 0 Thanked 9 Times in 8 Posts	Re: How not to run an A/B test Share on: To me, this type of article mentioning stats formulas scares off too many people from doing A/B testing. Much more important is how you generate test ideas - always use a test hypothesis, and always create insights from your web analytics tools and visitor feedback. Doing that will ensure you get better test results that get test significance much quicker, rather than getting bogged down by understanding statistics and potential 'analysis paralysis'. And don't forget, doing A/B testing is just a small piece of the CRO puzzle! And many sites don't even have enough traffic to do A/B testing - but they can still make improvements based on CRO best practices and do better analysis of their website and visitors.
	Rich Page http://www.rich-page.com \| Twitter: @richpage Author of 'Website Optimization: An Hour a Day'

8th Aug 2014, 12:44 PM	#3
jacobmark Master web marketer Join Date: 2010 Location: Fort Worth, Texas Posts: 25 Thanks: 4 Thanked 2 Times in 2 Posts	Re: How not to run an A/B test Share on: Google has a simple a/b split testing platform. You can test multiple different pages, you just need to pump a lot of traffics to the test pages.