Statistics to evaluate A/B-Tests

3 replies
Hi folks,

I hope someone can help me with my problem. I am currently running many a/b-tests and would like to know, if there is another measure for statistical significance that takes the value of each conversion into account? Let's say I am running an A/B-Test and increased my conversions by 12 % but reduced shopping baskets in the same time by 20 %. I often had the case now that I couldn't identify a significant difference between two versions but the shopping basket / Revenue seemed significantly different. Any idea, how to prove this statistically?

Thanks a lot!

Cheers
Julian
#a or btests #evaluate #statistics
Avatar of Unregistered
  • Profile picture of the author DABK
    When you do a/b testing with only one variable, the results speak clearly.

    I suggest you do that.

    Originally Posted by Julian Nik View Post

    Hi folks,

    I hope someone can help me with my problem. I am currently running many a/b-tests and would like to know, if there is another measure for statistical significance that takes the value of each conversion into account? Let's say I am running an A/B-Test and increased my conversions by 12 % but reduced shopping baskets in the same time by 20 %. I often had the case now that I couldn't identify a significant difference between two versions but the shopping basket / Revenue seemed significantly different. Any idea, how to prove this statistically?

    Thanks a lot!

    Cheers
    Julian
    {{ DiscussionBoard.errors[11698117].message }}
  • Profile picture of the author savidge4
    Originally Posted by Julian Nik View Post

    Hi folks,

    I hope someone can help me with my problem. I am currently running many a/b-tests and would like to know, if there is another measure for statistical significance that takes the value of each conversion into account? Let's say I am running an A/B-Test and increased my conversions by 12 % but reduced shopping baskets in the same time by 20 %. I often had the case now that I couldn't identify a significant difference between two versions but the shopping basket / Revenue seemed significantly different. Any idea, how to prove this statistically?

    Thanks a lot!

    Cheers
    Julian
    Lets set a baseline here just so I am sure I understand...

    100 visitors... you go from 6 conversions to 18 ( 12% increase )

    At the same time, you got 40 carts that create the 6 orders, and NOW you have 30 carts that create 18 conversions...

    Am I in the right line of thinking here?
    Signature
    Success is an ACT not an idea
    {{ DiscussionBoard.errors[11698140].message }}
  • Profile picture of the author savidge4
    Looking at this a second time...

    so lets get down to some basics... You are looking at 2 pages. Ill say its a lander and then the actual sales page.

    I believe what you are saying is you are making a change on the lander page, and the result is either (A) more hops to the offer but less conversion, OR (B) Less hops to the offer and more conversions.

    when you are making changes to a specific page, you have to analyze the effect of THAT page, and not how it is affecting the next step. From a lander page, you should only be concerned with how many hops are being made ( Shopping carts ).

    The next step is to then bring up the conversion of the actual offer page, to increase the total conversion or actual purchases.

    To try and get 2 birds with 1 stone will drive you round and round in circles.

    I am going to assume... you are either testing headlines, or the text in or around your call to action.

    Lets take a step back for a moment and think about the absolute basics in a working funnel ( Ad, Lander, Offer ) Ideally - or more so than ideally you WANT a consistent message across all 3 levels. If you are selling "10lb weight loss in 2 weeks" that message should carry across all 3 levels. If there is a break in that consistent message you will start to see the types of numbers you are seeing.

    Variant (A) - more hops less buys is probably textually in line with the ad, BUT your offer page deviates from that message - could be something as simple as weight loss in 2 weeks, vs weight loss in 14 days.

    Variant (B) is sending less traffic to the offer, but is converting better.

    Somewhere there is a simple break in consistent message. Could be the Ad, could be the Lander and it could be the Offer.

    Your AD is obviously working... so you want to match the headline and any other sales type text to what that ad is saying the person that clicks should expect. As in the headline for the lander should just about damn near REPEAT the message of the ad. An ad that says "lose 10 lbs in 2 weeks" should be met with a headline on the Lander "Lose 10 lbs in 2 weeks"

    Moving to the next step... If the lander says "Lose 10 lbs in 2 weeks" The product should clearly state "Lose 10 lbs in 2 weeks"

    Somewhere there is a disconnect in message that is not relating all the way back to the ad.

    And again, you are A/B testing a single element ( hopefully ) and should only be concerned with how that element is changing - IE going up or going down - and NOT how the next step is affected. You want to measure and test that SEPERATELY.

    Hope that Helps
    Signature
    Success is an ACT not an idea
    {{ DiscussionBoard.errors[11698222].message }}
Avatar of Unregistered

Trending Topics