When do sales figures become 'significant' ?

27 replies
If you made 2 sales a day of a product and then zero sales
then - obviously - you could not read anything into the
situation as the figures are too low to be of any
statistical significance.

Is it possible to say what a significant figure is ?


Harvey
#figures #ignificant #sales
  • Profile picture of the author John Taylor
    Harvey,

    The general rule of thumb is based on the
    number of actions taken: Number of sales,
    optins, downloads, etc.

    Assuming that you have X actions from your
    control copy, and Y actions from your test
    copy (where X is the better performing one).

    If the difference in the number of actions
    (X - Y) is greater than the square root of the
    total number of actions (X + Y), then the
    difference is statistically significant.

    It's generally thought that significance is
    achieved after 40 actions. But the above
    formula should be applied to be sure.

    John
    Signature
    John's Internet Marketing News, Views & Reviews: John Taylor Online
    {{ DiscussionBoard.errors[688728].message }}
    • Profile picture of the author Harvey Segal
      Hi John

      Thank you for the formula.

      With regard to your last statement

      It's generally thought that significance is
      achieved after 40 actions. But the above
      formula should be applied to be sure.
      do you actually mean
      - Don't look for significance until 40 actions. THEN apply the formula.

      Harvey
      {{ DiscussionBoard.errors[688766].message }}
      • Profile picture of the author John Taylor
        Harvey,

        Originally Posted by Harvey.Segal View Post

        do you actually mean
        - Don't look for significance until 40 actions. THEN apply the formula.
        No, significance can occur before then.

        However, it generally takes around 40
        actions. So, I'd suggest using it as a
        guideline threshold.

        John
        Signature
        John's Internet Marketing News, Views & Reviews: John Taylor Online
        {{ DiscussionBoard.errors[688777].message }}
    • Profile picture of the author eshannon
      Originally Posted by John Taylor View Post

      It's generally thought that significance is
      achieved after 40 actions. But the above
      formula should be applied to be sure.

      John
      40 conversions is hardly ever trustworthy. I've had dozens of tests show version A beating version B with 90%+ confidence interval after 40, 100, even 500 conversions..... only to flip flop totally at the 1000 conversion mark.

      I've been running high volume tests for years. The only time I truly trust a "winner" is if it's completely kicking the crap out of the other version after a few hundred conversions.
      Signature

      Eric Shannon

      {{ DiscussionBoard.errors[688913].message }}
      • Profile picture of the author John Taylor
        Originally Posted by eshannon View Post

        40 conversions is hardly ever trustworthy. I've had dozens of tests show version A beating version B with 90%+ confidence interval after 40, 100, even 500 conversions..... only to flip flop totally at the 1000 conversion mark.
        That's why I suggested to use it as a guideline and to
        use the formula to calculate statistical significance.

        John
        Signature
        John's Internet Marketing News, Views & Reviews: John Taylor Online
        {{ DiscussionBoard.errors[688925].message }}
        • Profile picture of the author eshannon
          Originally Posted by John Taylor View Post

          That's why I suggested to use it as a guideline and to
          use the formula to calculate statistical significance.

          John
          Right, and what I'm saying is that even the formulas can't be trusted with a sample as low as 40 conversions. (based on my own experience)
          Signature

          Eric Shannon

          {{ DiscussionBoard.errors[688999].message }}
  • Profile picture of the author Adaptive
    If the difference in the number of actions
    (X - Y) is greater than the square root of the
    total number of actions (X + Y), then the
    difference is statistically significant.
    This is very helpful, John, thanks for the mini math lesson.

    Regards,
    Allen

    P.S. I realize it's statistics but "mini math" has a zing to it from a copywriting point of view.
    {{ DiscussionBoard.errors[689113].message }}
    • Profile picture of the author Harvey Segal
      John

      How would you answer someone who says

      "I was making 2-3 sales a day then nothing for weeks so there's
      something funny going on"


      Harvey
      {{ DiscussionBoard.errors[698023].message }}
      • Profile picture of the author eshannon
        Originally Posted by Harvey.Segal View Post

        John

        How would you answer someone who says

        "I was making 2-3 sales a day then nothing for weeks so there's
        something funny going on"


        Harvey
        If you're talking testing, I'd tell that person to not even bother running a test, cause it would probably take a year to learn anything (unless one version REALLY kicked the crap out of the other).

        If you're just talking in general, tell the person to look at their analytics & the answer (if there is one) will be in there.
        Signature

        Eric Shannon

        {{ DiscussionBoard.errors[699467].message }}
        • Profile picture of the author Harvey Segal
          Originally Posted by eshannon View Post

          If you're talking testing, I'd tell that person to not even bother running a test, cause it would probably take a year to learn anything (unless one version REALLY kicked the crap out of the other).

          If you're just talking in general, tell the person to look at their analytics & the answer (if there is one) will be in there.
          Eric

          I'm not referring to testing but an ongoing sales situation.

          I'll express it another way.

          Suppose you made 2 sales as an affiliate in January and 0
          sales in February. Would you then suspect the affiliate
          company of cheating you out of your sales in February. Well,
          putting aside all mathematical formulas, common sense says
          it would be ridiculous to base any conclusion on such a low
          number of sales.

          Now move to another situation. Someone claims 2-3 sales a
          day then nothing for weeks. Does this fall into the same
          category i.e sales too low. If so is there any (approximate)
          figure you would suggest ?

          Harvey
          {{ DiscussionBoard.errors[699887].message }}
          • Profile picture of the author eshannon
            Originally Posted by Harvey.Segal View Post

            Eric

            I'm not referring to testing but an ongoing sales situation.

            I'll express it another way.

            Suppose you made 2 sales as an affiliate in January and 0
            sales in February. Would you then suspect the affiliate
            company of cheating you out of your sales in February. Well,
            putting aside all mathematical formulas, common sense says
            it would be ridiculous to base any conclusion on such a low
            number of sales.

            Now move to another situation. Someone claims 2-3 sales a
            day then nothing for weeks. Does this fall into the same
            category i.e sales too low. If so is there any (approximate)
            figure you would suggest ?

            Harvey
            Ahh ok, I gotcha. LB laid it out pretty well in the last post. This isn't really a question of statistical significance. It's more a matter of your affiliate not understanding how to use analytics. Ask him if he's got GA installed, and if so, is he tracking a click to the merchant site as a goal metric. If not, then he's out of luck. But if he is, then you can start digging down into traffic sources and maybe find a problem.

            But again, a site with 2-3 sales a day won't have much traffic, so it's harder to get consistent trends.
            Signature

            Eric Shannon

            {{ DiscussionBoard.errors[700418].message }}
            • Profile picture of the author Harvey Segal
              Ok let's try to put a figure on this.

              Suppose we get a constant 250 visitors every day with
              a conversion rate of 1% - giving us the 2-3 sales per day.

              When this drops to zero do we say
              - no problem, law of averages
              or
              - let's check it out

              We'll assume that we are not selling John's
              $10,000 products just a humble ebook


              Harvey
              {{ DiscussionBoard.errors[700611].message }}
              • Profile picture of the author LB
                Originally Posted by Harvey.Segal View Post

                Ok let's try to put a figure on this.

                Suppose we get a constant 250 visitors every day with
                a conversion rate of 1% - giving us the 2-3 sales per day.

                When this drops to zero do we say
                - no problem, law of averages
                or
                - let's check it out

                We'll assume that we are not selling John's
                $10,000 products just a humble ebook


                Harvey
                Yep, I would check it out. I think most people would...

                The question on what to check out depends on the traffic sources of course.
                Signature
                Tired of Article Marketing, Backlink Spamming and Other Crusty Old Traffic Methods?

                Click Here.
                {{ DiscussionBoard.errors[700870].message }}
                • Profile picture of the author Martin Avis
                  Harvey,

                  There are so many variables that I don't think you can ever really pin down an accurate answer.

                  For example, just the 250 visitors per day is open to a vast array of influences:

                  Where they are attracted from
                  If natural search are they from a #1 position or a #5
                  What time of day
                  What day of week
                  How close it is to payday
                  How close it is to tax day
                  What is on the news
                  and on and on.

                  You can easily have your 250 being made up of a wildly different demographic each day.

                  Or even if the demographic is stable, its behavior may fluctuate.

                  And then there are external factors:

                  The economy
                  The weather
                  Other products ebbing and lowing in popularity
                  Negative or positive reviews
                  New launches in your market or related markets


                  And then, closer to home there are other issues to contend with:

                  Your server's uptime
                  Your host's speed
                  Your site's compatibility with different browsers
                  Your payment processor's speed of page loading

                  I run a very big campaign for a major client, using banner ads to drive traffic to a video page. We noticed that suddenly the number of people clicking on the banners resulted in far fewer arriving at our video page than expected (there is never a 100% correlation because some people will always abandon an action before the page loads, but usually less than 5%)

                  The change was significant - it appeared that up to 70% of people who clicked the ad never arrived at the desired landing page.

                  I buy tens of millions of impressions each month, so the stats were very significant indeed!

                  After a great deal of analysis with some very sophisticated tracking software we found that the cause was a very small change that had happened at the server level that resulted in the video page taking, on average, 4 seconds longer than normal to load.

                  To those of us monitoring the account we had never noticed the slowdown because our broadband connections are fast and the 'delay' we personally experienced was much less. But couple the fact that many, many people are on sub optimal Internet connections and all of us have evolved to have teensy-weensy attention spans and the 4 second average delay resulted in massive page abandonment. And a loss of 70% of our business.

                  Needless to say we tracked down the problem and put it right, and the figures jumped back to normal.

                  Small things can have big effects - but tracking down which small things is the stuff of late nights and chaos theory!

                  Martin
                  Signature
                  Martin Avis publishes Kickstart Newsletter - Subscribe free at http://kickstartnewsletter.com
                  {{ DiscussionBoard.errors[701103].message }}
                • Profile picture of the author Harvey Segal
                  Martin (Avis) and others.

                  Your points about why sales might decline are excellent
                  and could be the basis of an article or even ebook of its own.

                  However I still think you are talking about Step 2
                  of the process where Step 1 is the basic question

                  "Just knowing these figures alone is it worth proceeding to Step 1
                  - to find out the reasons"


                  Here is one answer
                  Originally Posted by LB View Post

                  Yep, I would check it out. I think most people would...

                  Harvey
                  {{ DiscussionBoard.errors[701544].message }}
  • Profile picture of the author John Taylor
    Harvey,

    There are so many different factors to take into
    account it's impossible to answer the question.

    If my sales had dwindled or disappeared, I'd begin
    by looking at my referral stats to see if the source
    of traffic was consistent.

    I've actually experienced a complete loss of sales
    of one of my products for two weeks. It turned out
    that my best affiliate had stopped all his PPC
    campaigns due to a period in hospital. The best
    analytics in the world won't explain that one!

    John
    Signature
    John's Internet Marketing News, Views & Reviews: John Taylor Online
    {{ DiscussionBoard.errors[698043].message }}
    • Profile picture of the author Harvey Segal
      John

      I realise it's difficult to answer but, if we assume traffic
      is consistent, I thought from the above discussion you would say

      "until sales reach a level of 40 per day you can't necessarily make a judgement"

      - or am I misintrepreting what has been said ?


      Harvey
      {{ DiscussionBoard.errors[698073].message }}
  • Profile picture of the author derekwong28
    If you want to do it properly, you will need to plot your sales out on a Shewhart Control Chart as they do in factories and laboratories for quality control purposes.

    Control chart - Wikipedia, the free encyclopedia


    You will need to work out your usual 2 s.d. and 3 s.d. first.

    Westgard rules are commonly used to interpret the results. The most commonly used rules are.

    Warning Rules

    The control value exceeds the mean by 2 s.d. (p<.005)
    Two consecutive values exceed the target value on the same side of the mean by 2 s.d. This may indicate the presence of systematic errors.
    Four consecutive control values exceed the same limit. This again indicates the possible presence of systematic errors and may indicate the need to perform instrument maintenance or equipment calibration.

    Mandatory rules

    The control value exceeds the target value by 3.s.d. (p<0.01:99.7% of values should lie within 3 s.d.) This indicates that the assay run is out of control
    Where the IQC sample is tested in duplicate, the difference in SD between the duplicates should not exceed 4 SD
    10 consecutive values are on the same side of the mean or target value. This detects systematic errors. This may happen when a new test batch or introduced or changes in the calibration of equipment.

    Basically, when a warning rule is breached, it tells you that potentially something is wrong and you should investigate.. If a mandatory rule is breached, then something is almost definitely wrong.


    Derek
    Signature

    Do not get between a wombat and a chocolate biscuit; you will regret it dearly!

    {{ DiscussionBoard.errors[698058].message }}
  • Profile picture of the author John Taylor
    Harvey,

    It's not 40 sales per day. It's 40 actions over the
    period of testing.

    In the context of your original post, then my answer
    would be:

    Looking at a small batch of data in isolation is not
    going to give you accurate results, unless that data
    is in itself statistically significant.


    I'd probably also suggest that they look for spikes in
    the data where they may have been seeing 4-5 sales
    per day. But people never seem to complain about
    above average sales figures. ;-)

    John
    Signature
    John's Internet Marketing News, Views & Reviews: John Taylor Online
    {{ DiscussionBoard.errors[698186].message }}
  • Profile picture of the author John Taylor
    Harvey,

    There isn't an approximate figure. Your test
    results are either statistically significant or
    they are not.

    I've probably done you a disservice by offering
    you a rule of thimb guideline of 40 actions.

    However, as you mention above, common sense
    should help you to see if your results require
    investigation.

    If your sales drop from "2-3 per day to nothing
    for weeks" then, as I said earlier, I'd look at my
    stats to identify anything that may have changed.

    John
    Signature
    John's Internet Marketing News, Views & Reviews: John Taylor Online
    {{ DiscussionBoard.errors[699912].message }}
    • Profile picture of the author Harvey Segal
      Originally Posted by John Taylor View Post

      If your sales drop from "2-3 per day to nothing
      for weeks" then, as I said earlier, I'd look at my
      stats to identify anything that may have changed.
      Therefore can I deduce from this that of the 2 cases

      1. 2 sales January, 0 sales in February - not worth investigating
      2. 2- 3 sales per day then nothing for weeks - worth investigating

      Harvey
      {{ DiscussionBoard.errors[699925].message }}
      • Profile picture of the author John Taylor
        Harvey,

        Originally Posted by Harvey.Segal View Post

        Therefore can I deduce from this that of the 2 cases

        1. 2 sales January, 0 sales in February - not worth investigating
        2. 2- 3 sales per day then nothing for weeks - worth investigating

        Harvey
        On the face of it that would seem to be a
        reasonable assumption. However, it depends
        upon the context.

        If those 2 sales in January were each of the
        value of $10,000, I'd damn well investigate
        why I wasn't selling anything in February.

        Your sales figures and conversion rates will
        only tell you what happened. It's up to you
        to decide if you want to know the reasons
        why.

        John
        Signature
        John's Internet Marketing News, Views & Reviews: John Taylor Online
        {{ DiscussionBoard.errors[699943].message }}
    • Profile picture of the author Martin Luxton
      Harvey,

      Putting together John's example of the hospitalized affiliate and Nassim Nicholas Taleb's "The Black Swan" puts another spin on this.

      Even if there is a significant change do you really know what it can be attributed to?

      Taleb argues that past performance can often bear no relevance to future results (the Problem of Inductive Knowledge). His classic example is the Turkey which believes every day is spent being fed and pampered until the day before Thanksgiving forces a revision of belief.

      He also says "We just do not know how much information there is in the past."

      In essence, you do not know exactly why you were getting those 2 regular daily sales so would you also know why they stopped?

      Martin
      {{ DiscussionBoard.errors[699958].message }}
  • Profile picture of the author LB
    In essence, you do not know exactly why you were getting those 2 regular daily sales so would you also know why they stopped?
    Excellent point.

    Most people have no idea where their sales are coming from.

    An answer like "PPC" is insufficient. The same PPC campaign can vary drastically in its traffic and if it's a content network campaign then you'd better be tracking where the ads are showing. Maybe a competitor moved in, listed a PPC ad above mine and is offering bonuses that I'm not.

    If you are blindly slapping up articles and praying for the best as so many do then a sudden fall in sales is almost to be expected. The "Google dance" often throws new content high in the SERPs resulting in a quick spurt in sales until the page "settles" a few pages deep. This is often why article marketers get stuck churning out articles every day to keep sales coming in.

    Ultimately, 2-3 sales per day (60-90 per month) then moving to 0 would definitely give me concern. However, without traffic figures, referrers, etc. It's meaningless.

    i.e. If I sent 10,000 visitors to a site and made 90 sales one month but then the next month I only sent 400 visitors...is the 0 really that unreasonable?

    And finally, I don't know who is supplying the info but affiliates often, um...exaggerate.

    "I was making three sales a day for weeks and now nothing for weeks!"

    Translation: "2 weeks ago I made 10 sales and thought I would be a millionaire but this week no sales came in. I'm not tracking anything."

    Lastly, some perspective.

    Last Wednesday I had 50 sales, this Wednesday I had 102. That's a variation of 52 sales. Next Wednesday it might be 20. In my world "nothing changed". It's normal fluctuation from where I sit.

    2-3 sales per day is significant...but perhaps not enough significant data to come to any solid conclusions.
    Signature
    Tired of Article Marketing, Backlink Spamming and Other Crusty Old Traffic Methods?

    Click Here.
    {{ DiscussionBoard.errors[700000].message }}
  • Profile picture of the author Dmitry
    Harvey - you say everyday... but for what period of time?

    If someone's getting 2 sales and 250 visitors a day for let's
    say a year and then it all suddenly drops for a few weeks -
    then that looks more significant to me than let's say 2 sales
    and 250 visitors for a period of a month and then nothing for
    a few weeks.
    {{ DiscussionBoard.errors[701731].message }}
    • Profile picture of the author Harvey Segal
      Originally Posted by Dmitry View Post

      Harvey - you say everyday... but for what period of time?

      If someone's getting 2 sales and 250 visitors a day for let's
      say a year and then it all suddenly drops for a few weeks -
      then that looks more significant to me than let's say 2 sales
      and 250 visitors for a period of a month and then nothing for
      a few weeks.
      Excellent point.

      Maybe someone can offer a guide as to what would be
      reasonable time periods for the before and after events.


      Harvey
      {{ DiscussionBoard.errors[701803].message }}

Trending Topics