Kelvin Chan

Hi Warriors,

It’s been a while since I last shared & thought I’ll write to share a MVT test I’m performing at the moment…

A MVT test is a more complex version of a simple A/B test (both of which I’ll be explaining in a short while’s time…)

This sharing is INTENDED to stimulate hunger to improve us all as marketers and while things aren’t conclusive, the early findings are pretty interesting and might prove useful to some reading this. In that spirit, I’ll proceed to share. (On a PERSONAL level, this writing serves as a journaling record)

First things first, I’m going to assume you know what a squeeze or lead capture page is and its CHIEF purpose (i.e. to captivate your intended target audience and to capture their true email address for future marketing communications).

To explain what a MVT test is, it’s wise to start off with its simpler version - the A/B test.

To provide an example and an analogy, let’s use popcorns!

Suppose everyone likes popcorns and there are 2 types - SWEET popcorn and SALTY popcorn.

In a sample size of 1000 people, if 600 prefers SWEET popcorn and 400 SALTY popcorn, then SWEET popcorn is declared the winner.

Simple?

With a MVT test, there is more than 1 element of which you can have any number of variations for each of those elements.

Keeping to our popcorn example, let’s say these are the 2 elements & its 2 variations:
1) Preferred taste - SWEET or SALTY
2) Preferred container - CUP or BOWL

And if the results were as follows…
Combination #1: SWEET popcorn in a CUP (e.g. 400 people)
Combination #2: SWEET popcorn in a BOWL (e.g. 200 people)
Combination #3: SALTY popcorn in a CUP (e.g. 300 people)
Combination #4: SALTY popcorn in a BOWL (e.g. 100 people)

Hence, the WINNER is - SWEET popcorn in a CUP.

Now…

If you’re the kind who loves working on details, read on till the end… I trust it will be well worth your time.

If you’re the sort who can’t be bothered about details and couldn’t care less, I’ll leave you to decide what’s best for yourself.

Moving right along…

In my current MVT test, I’ve got 4 elements:
1) Top bar with logo (PRESENT or ABSENT)
2) Headline (RED or BLACK)
3) CTA text (With the word “FREE” or without)
4) Privacy statement (PRESENT or ABSENT)

This means that we have a total of 16 unique permutations / combinations (ask your math teacher if necessary…lol!)

Before we go on, I’ll like you to…
+ Predict which of the 4 elements impacts conversions the most (AT this early point in time…)
+ Predict which variation of each of the 4 elements is the winner (Again, AT this early point in time…)

Ready? =)

Here are initial results based on 437 visitors so far…



As it stands…
+ The element that impacts conversions the MOST is… the Privacy statement (Did you get this right?). Removing it helps conversions (so far…)!
+ Having the word “FREE” in the call-to-action button helps boost conversions
+ The top performing combination (of the 16) is converting at 71.43%, the worst is at 33.33% and the overall at 50.11% (Not represented by the image above)

In this current phase of MVT, a total of 300 to 500 visitors will go to each of the 16 combinations requiring a minimum of 4800 to 8000 visitors. (Can’t wait to see what the final results are - where I’ll then eliminate the weakest combinations and further tweak the strongest ones)

Prior to this phase, simple tests were done to determine which squeeze page design to test with.

Once this current phase of MVT is over (results of which will then be taken with a lot more WEIGHT compared to now), the next phases will test out:
+ Squeeze Page Headlines (16 variations)
+ Pre-Headlines (Presence v.s. Absence)
+ Sub-Headlines (Presence v.s. Absence)
+ CTA text, color & perhaps font? (More variations)
+ Graphical Headline v.s. Textual Headline (With same EXACT copy)

These tests will require about 40,000 to 50,000 visitors over the next couple of weeks… (Should be a fascinating time for me!)

Meantime, feel free & encourage you to… minimally do a simple A/B split test (if not a MVT test) based on my early findings.

On this note, 1 tip… make sure to do your tests based on a variety of traffic sources (my current phase requiring 8000 visitors will come from at least 20 different sources).

Trust you’ve enjoyed this and feel free to share your own findings (in the past or going forward) too!

*Scurries off to check on the latest MVT results* =D

ProaSailor

Hi Kelvin,
It is indeed satisfying to see that you can find a 2X change in conversion rate based on your multivarite test (33% to 71%).
Question 1: Are you familiar with Design of Experiments [DoE] and more specifically, factorial designs?
These are statistical tools that are widely used in agriculture and engineering to optimize things like crop yield and manufacturing yield (% product that meets quality specs).
I am new to virtual marketing and direct response marketing, but I have a lot of experience and training in Design of Experiments.
This kind of optimization looks like a good niche to get into - for information products and services.
Question 2: Would you say that the vast majority of marketers use A/B split testing, while only a small minority know about multivariate testing?
I would love to chat about the market potential for this kind of thing.
Best regards,
Doug Weir

savidge4

I would say that A/B testing is without question the majority of testing that takes place. I personally Do far more A/B testing than I do multi variant. Multi Variant as I see it does have a place in a testing strategy, but I think you are cutting your results short if this is the only testing you are doing.

I would with out question back up multi variant testing with a more targeted A/B test strategy to ensure that each element on its own is working at its optimal.

A good example of this concept would be in this thread: http://www.warriorforum.com/conversi...sion-rate.html if you look at all of the suggested changes that I made in post #2 you could very quickly make the changes and of course then test this to the original IE a multi variant test. You then would want to go in and test many of these element on their own to get maximum effect.