Conversion Data Gurus - Multivariate or A/B Split

I go with incremental A/B testing. When I get a large sample size, I test for statistical significance. If it exists, then I remove the weaker copy (obviously) and replace it with new copy.

I do this until I feel I've got a control (how many times depends on the importance of the project).

I feel multivariate is too large of an investment in the copy. Your time is likely better spent in other places (should a copywriter be saying this?)

you dont multi variant test new stuff..... you test a/b first.

THEN drill down when you have a winner

Iterative bayesian split testing.

Frequentist split testing (confidence interval testing) is fundamentally flawed and costs the marketer more money than is necessary, and does not answer the essential question of which version is better. Currently there are only two bayesian testers out there - one of which I built - but it's a little buggy since I only spent one weekend on it (it works with "normal" inputs - and if you get a calculation the results are sound - but it blows up with extremely large inputs). Eventually, all split testing will be iterative bayes. I know that doesn't make any sense without context, but it's a long story. Read "The Signal and the Noise," by Nate Silver.

Split Testing Application

morning dave.

So, I found this...

An Intuitive (and Short) Explanation of Bayes’ Theorem | BetterExplained

But I'm still not really sure how this relates to direct response, as the results we're looking at are pretty simple (which one sells more).

Put another one, in our case, aren't the test results the REAL results?

I'm interested to learn more if anyone has any input.

-Daniel

I'll try to do a longer write-up when I have time, but here's another reasonable link:
8 Flaws in A/B Split Testing | Iterative Path

The simplest answer is to stick to bayesian A/B testing, and avoid multi-variable testing (and I was trained in Taguchi testing, so I don't say that lightly).

Here is the only other bayesian split-test tool on the web besides my own:

AB split test graphical Bayesian calculator

Here you go. Nice and easy for us meatheads to understand

Bayesian statistics for dummies


Kevin Boone's Web site

Solid thread, gang.

Fargin aye... maybe we're getting back some mojo here?

Awesome, thanks guys.

I've never been a fan of multivariate testing anyway - always A/B splits, but this is a cool new tool I'll definitely be using from now on.

-Daniel

Lot of conversion tools and recommendations here (via Gary Zaydman):
Q&A: Ben Jesson on conversion rate optimisation | Econsultancy

The site they analyze in the post is here:
CAR LEASING CHEAP LEASE CARS SALES CAR LEASING CHEAP BUSINESS HIRE DEALS

Man, seriously...thanks for all the insight guys. Apparently I have some quick knowledge consumption on the plate...will tear through it and make a decision and begin. 2 thumbsup

While I hear you....A/B and some of it's variants seem the simplest route to take...and effective.

I"m so damned lured into wanting to run testing on multiple elements on the page as I know sometimes goofy little shit like color of buttons or shaded backgrounds in varying colors, etc can make a diff...

I will likely end up with some simple A/B stuff utilizing some Google Goals + the A/B Max plugin as I typically run everything on WP.

Although this is tempting the shite out of me:

WP Test Monkey | Multivariate Testing for WordPress



Thanks again...you f0ckers rock.

Yeah - I really have to agree with Bruce here. I've done hundreds of split tests. You know what really makes a big difference? Traffic sources. Make sure your copy targets a real urgent problem, and you're getting in front of an audience with that pain point. That is the difference between a campaign that works and one that doesn't.

Split testing background colors will never, ever turn a campaign from losing to winning. When people say that adding a guarantee seal gives a 25% bump, you have to consider you might be going from .8% to 1%, which is pretty big. But if you're at .1, you're only going to go from .1% to .125% which is still a lifeless campaign. Changing traffic sources could get you from .1 to .8 overnight which is 800%.

Don't test page elements. Test traffic sources.

Great stuff Bruce and Dan...thanks for the reminder about the 'not testing whispers'...it's been a while, but I've heard something like that before.

And yes...will get myself unstuck on this issue right now...simply go for A/B split and focus on traffic testing. Kick ass reminders.
You guys rock.