23 {{ upvoteCount | shortNum }}
23 {{ upvoteCount | shortNum }}

Conversion Data Gurus - Multivariate or A/B Split

by SeanIM
15 replies
Which method do you prefer to rock when testing out new offers?

And which tools do you like to use?

Thanks in advance...
#a or b #conversion #data #gurus #multivariate #split
  • Profile picture of the author verial
    Originally Posted by SeanIM View Post

    Which method do you prefer to rock when testing out new offers?

    And which tools do you like to use?

    Thanks in advance...
    I go with incremental A/B testing. When I get a large sample size, I test for statistical significance. If it exists, then I remove the weaker copy (obviously) and replace it with new copy.

    I do this until I feel I've got a control (how many times depends on the importance of the project).

    I feel multivariate is too large of an investment in the copy. Your time is likely better spent in other places (should a copywriter be saying this?)
    Signature
    {{ DiscussionBoard.errors[7899854].message }}
  • Profile picture of the author davemiz
    you dont multi variant test new stuff..... you test a/b first.

    THEN drill down when you have a winner
    Signature

    “Judge your success by what you had to give up in order to get it.”
    ― Dalai Lama XIV

    {{ DiscussionBoard.errors[7900316].message }}
  • Profile picture of the author dancaron
    Iterative bayesian split testing.

    Frequentist split testing (confidence interval testing) is fundamentally flawed and costs the marketer more money than is necessary, and does not answer the essential question of which version is better. Currently there are only two bayesian testers out there - one of which I built - but it's a little buggy since I only spent one weekend on it (it works with "normal" inputs - and if you get a calculation the results are sound - but it blows up with extremely large inputs). Eventually, all split testing will be iterative bayes. I know that doesn't make any sense without context, but it's a long story. Read "The Signal and the Noise," by Nate Silver.

    Split Testing Application
    Signature
    MemberMouse - The WordPress Membership Plugin Trusted by 7 Figure Marketers
    {{ DiscussionBoard.errors[7900356].message }}
  • Profile picture of the author dancaron
    morning dave.
    Signature
    MemberMouse - The WordPress Membership Plugin Trusted by 7 Figure Marketers
    {{ DiscussionBoard.errors[7900394].message }}
  • Profile picture of the author Daniel Scott
    So, I found this...

    An Intuitive (and Short) Explanation of Bayes’ Theorem | BetterExplained

    But I'm still not really sure how this relates to direct response, as the results we're looking at are pretty simple (which one sells more).

    Put another one, in our case, aren't the test results the REAL results?

    I'm interested to learn more if anyone has any input.

    -Daniel
    Signature

    Always looking for badass direct-response copywriters. PM me if we don't know each other and you're looking for work.

    {{ DiscussionBoard.errors[7902753].message }}
  • Profile picture of the author dancaron
    I'll try to do a longer write-up when I have time, but here's another reasonable link:
    8 Flaws in A/B Split Testing | Iterative Path

    The simplest answer is to stick to bayesian A/B testing, and avoid multi-variable testing (and I was trained in Taguchi testing, so I don't say that lightly).

    Here is the only other bayesian split-test tool on the web besides my own:

    AB split test graphical Bayesian calculator
    Signature
    MemberMouse - The WordPress Membership Plugin Trusted by 7 Figure Marketers
    {{ DiscussionBoard.errors[7903313].message }}
  • Profile picture of the author Bruce Wedding
    Here you go. Nice and easy for us meatheads to understand

    Bayesian statistics for dummies


    Kevin Boone's Web site
    Signature
    {{ DiscussionBoard.errors[7903355].message }}
  • Profile picture of the author BrianMcLeod
    Solid thread, gang.

    Fargin aye... maybe we're getting back some mojo here?
    Signature
    Brian McLeod
    {{ DiscussionBoard.errors[7903402].message }}
  • Profile picture of the author Daniel Scott
    Awesome, thanks guys.

    I've never been a fan of multivariate testing anyway - always A/B splits, but this is a cool new tool I'll definitely be using from now on.

    -Daniel
    Signature

    Always looking for badass direct-response copywriters. PM me if we don't know each other and you're looking for work.

    {{ DiscussionBoard.errors[7903405].message }}
  • Profile picture of the author KingOfContentMarketing
    Lot of conversion tools and recommendations here (via Gary Zaydman):
    Q&A: Ben Jesson on conversion rate optimisation | Econsultancy

    The site they analyze in the post is here:
    CAR LEASING CHEAP LEASE CARS SALES CAR LEASING CHEAP BUSINESS HIRE DEALS
    Signature
    Marketing is not a battle of products. It is a battle of perceptions.
    - Jack Trout
    {{ DiscussionBoard.errors[7903612].message }}
  • Profile picture of the author SeanIM
    Man, seriously...thanks for all the insight guys. Apparently I have some quick knowledge consumption on the plate...will tear through it and make a decision and begin. 2 thumbsup

    While I hear you....A/B and some of it's variants seem the simplest route to take...and effective.

    I"m so damned lured into wanting to run testing on multiple elements on the page as I know sometimes goofy little shit like color of buttons or shaded backgrounds in varying colors, etc can make a diff...

    I will likely end up with some simple A/B stuff utilizing some Google Goals + the A/B Max plugin as I typically run everything on WP.

    Although this is tempting the shite out of me:

    WP Test Monkey | Multivariate Testing for WordPress



    Thanks again...you f0ckers rock.
    Signature

    >>> The 3 Secrets to How I Made $152,414 in 7 Months! <<<

    {{ DiscussionBoard.errors[7903738].message }}
    • Profile picture of the author Bruce Wedding
      Originally Posted by SeanIM View Post

      I"m so damned lured into wanting to run testing on multiple elements on the page as I know sometimes goofy little shit like color of buttons or shaded backgrounds in varying colors, etc can make a diff...
      I have a book I keep in my "library" which is perfect for the smallest room in my house because it is filled with 2,239 gems for direct marketing It's called, "2,239 Test Secrets for Direct Marketing Success".

      It's filled with comments from industry leaders on every aspect of DM. In the chapter on testing, here are a few you should take to heart:

      Lew Smith's Precept
      The success of any direct marketing effort depends on three things: Product, Offer and lists.
      Everything else - creative, paper stock, format- is secondary and tertiary.
      Ed Mayer's Corollary
      Don't test whispers.
      Test $9.95 vs $9.98 or blue paper vs pink paper or Garamond vs Times Roman only AFTER you have exhausted every other possible test.
      But the most important, IMHO, by far is this one...

      Anver Suleiman's Observations

      Let your competitors test for you.
      Don't test stuff your competition has already tested, especially the big competitors. If they tried a certain approach and dropped it, assume it didn't work. And if they are using something over and over, assume it does work. No need to test and reinvent until your model proves itself.
      So how do you do that? Use services like What Runs Where and AdBeat. If you're on a budget, there's a great free service out there now too but I'm not blasting it out in public. If you want to know about it, PM me.

      Analyze the top CB products, etc. Do your research.
      Signature
      {{ DiscussionBoard.errors[7905120].message }}
  • Profile picture of the author dancaron
    Yeah - I really have to agree with Bruce here. I've done hundreds of split tests. You know what really makes a big difference? Traffic sources. Make sure your copy targets a real urgent problem, and you're getting in front of an audience with that pain point. That is the difference between a campaign that works and one that doesn't.

    Split testing background colors will never, ever turn a campaign from losing to winning. When people say that adding a guarantee seal gives a 25% bump, you have to consider you might be going from .8% to 1%, which is pretty big. But if you're at .1, you're only going to go from .1% to .125% which is still a lifeless campaign. Changing traffic sources could get you from .1 to .8 overnight which is 800%.

    Don't test page elements. Test traffic sources.
    Signature
    MemberMouse - The WordPress Membership Plugin Trusted by 7 Figure Marketers
    {{ DiscussionBoard.errors[7905234].message }}
  • Profile picture of the author SeanIM
    Great stuff Bruce and Dan...thanks for the reminder about the 'not testing whispers'...it's been a while, but I've heard something like that before.

    And yes...will get myself unstuck on this issue right now...simply go for A/B split and focus on traffic testing. Kick ass reminders.
    You guys rock.
    Signature

    >>> The 3 Secrets to How I Made $152,414 in 7 Months! <<<

    {{ DiscussionBoard.errors[7905357].message }}

Trending Topics