Tell me if I'm right or wrong about A/B testing and Multivariate testing.

8 replies
I'm learning digital marketing at the moment. through my studying I come across A/B studying and Multivariate test. Please letcme know Irvin wrong or not. I'm more shaky on the multivariate testing then A/B testing.

A/B testing is when you make two varrients of a website and you keep one original and the other one you change something to see the response and conversion rate of the two. You then go with the one with the most positve repsonse.

Mulitivariate is when you dont make two varrient of a website and just change something off that one website.

Please respond.
#a or b #multivariate #wrong
  • Profile picture of the author ChrisBa
    I use a/b testing and it looks like you have it right. Keep in mind after your first split test and you have the results again you should change something else and test again.
    {{ DiscussionBoard.errors[11247075].message }}
  • Profile picture of the author Jason Kanigan
    Remember high school science?

    How many variables did you change at at time to test?

    One.

    Some time in 2008 I was zipping along making changes to the company website of my employer at the time, and went over to ask a question of the lead programmer. He reminded me of this key fact to slow me down.

    If you make multiple changes, and something actually works to get you better results...how do you duplicate it? You don't know what you did that made the improvement.

    One thing at a time, and give it enough time to work.
    {{ DiscussionBoard.errors[11247078].message }}
    • Profile picture of the author ChrisBa
      Originally Posted by Jason Kanigan View Post

      Remember high school science?

      How many variables did you change at at time to test?

      One.

      Some time in 2008 I was zipping along making changes to the company website of my employer at the time, and went over to ask a question of the lead programmer. He reminded me of this key fact to slow me down.

      If you make multiple changes, and something actually works to get you better results...how do you duplicate it? You don't know what you did that made the improvement.

      One thing at a time, and give it enough time to work.
      Agreed. Test one at a time. But don't stop after getting the results of the first run. (IMO at least. )
      {{ DiscussionBoard.errors[11247629].message }}
  • Profile picture of the author JohnMcCabe
    You have A/B testing figured out, with one exception. You don't just take the more favorable result. You need something called "statistical significance". That means that the result of the test can be attributed to the difference in the two variants, not just random chance. There are a lot of calculators online that will let you put in the number of responses to each version and tell you if the difference is significant, and to what degree.

    Aim for results with 85-90% significance. Getting to 99% is overkill for a marketing project.

    Multivariate testing, as the name implies, tests multiple versions of a page (in this case), swapping in multiple versions of a variable (Headline, set of bullets, offer, images, etc.). This is much more complex.

    With A/B testing, you have two possibilities, A or B.

    With multivariate testing you multiply the number of versions of each variable. Say you have 3 versions of 3 different variables. You end up with 3x3x3, or 27 possible combinations. Needless to say, you need a whole heck of a lot more trials to get a significant answer.

    I played with MV testing a bit, and I gave it up because of the complexity. And I have a BS in engineering.
    {{ DiscussionBoard.errors[11247314].message }}
    • Profile picture of the author Terrence Morrow
      Ok I understand a lot more now. I apreaciate it and thank you.
      {{ DiscussionBoard.errors[11247420].message }}
  • Profile picture of the author pauloadaoag
    Administrator
    Just to add to what JohnMcCabe said, you will also need tons more traffic to reach statistical significance if you go down the MV route.
    Signature
    {{ DiscussionBoard.errors[11247609].message }}
  • Profile picture of the author Maxxx333
    this is exactly what is A/B testing, but let's be honest, you should not stop the test as soon as you have any better results, but still change some few things, and see how the results change. You will only stop when the results will start fading up!
    {{ DiscussionBoard.errors[11248702].message }}

Trending Topics