How long do you test a method before dropping it?

6 replies
When trying a new method or strategy, how long do you usually give it before deciding it's not worth continuing?
Do you set a fixed testing period or base it on specific metrices?
#dropping #long #method
Avatar of Unregistered
  • Profile picture of the author Frank Donovan
    That's impossible to answer without knowing the context. You'll need to provide more details - are we talking about a headline or copy revamp? A new marketing platform? Some changes show immediate results, others can take weeks, months, even years to manifest.

    The testing period and metrics you measure will depend on what you're trying to test and what you want to achieve.
    Signature

    {{ DiscussionBoard.errors[11844494].message }}
    • Profile picture of the author JD Goff
      Oooh, I like this question because Ive been there before.

      This depends on the method. Some stuff gives fast feedback (ads, outreach, affiliate promos), and some stuff is slow (SEO, content, email list building). For quick feedback methods I'd say 7-14 days is enough to know if the angle is working.
      For slower methods I try to give it 30 days of consistent execution before I decide it's dead.

      I think most people quit way to early, or they test somethings while only doing it halfway. Biggest thing for me is tracking some kind of metric (clicks, opt-ins, replies, engagement) so that you're not guessing. I hope that helps and good luck.
      {{ DiscussionBoard.errors[11844519].message }}
  • Profile picture of the author Jason Kanigan
    Originally Posted by felix1232 View Post

    When trying a new method or strategy, how long do you usually give it before deciding it's not worth continuing?
    Do you set a fixed testing period or base it on specific metrices?
    I took 3 years of statistics in college and did pretty well with it. Fact is, people say they're doing "split testing" but they have nowhere near enough instances (called the n number) of the thing to find any statistical relevancy..

    There is a myth that an n of 30 is enough. I don't agree, unless it's to determine something like "will a hammer smash an egg?.

    First, I would gather at least a n of 100 before making a decision about whether a thing works or not.

    Second, you have to be very careful to ensure you are treating your observations like a high school science experiment. This means changing ONE variable at a time and then waiting for that good n number of results to come in.

    The problem is, people muddle. Their sales page doesn't work, so simultaneously they change the color, the headline, the CTA button, and add a video. Now, even if results do improve, you have no idea what actually did it and can't replicate that outcome.

    Third, if you are learning a skill, like how to run paid ads to bring in at least somewhat prequalified leads, or how to sell using a specific process, then you are going to have to do many repetitions and probably get some coaching to get good at it. This is not a simple thing. You may be doing something wrong but not see it. I would expect at least 90 days with lots of repetition (aka 'reps') and feedback to learn quickly.

    So if your n=100 and you are getting one observation in a day, it's going take you 4 months to collect enough data to know anything for sure.

    I can tell you having helped people build businesses since 2012, and watched a lot of people who basically approached their new business idea like it was a New Years Resolution, most people give up after just three days of sustained effort. Consequently, they learn nothing and struggle year after year. Sad, but true.
    {{ DiscussionBoard.errors[11844499].message }}
  • You illah or wellah bcs this?

    (Do naht ask Moi how I split tested the fkr.)
    Signature

    Lightin' fuses is for blowin' stuff togethah.

    {{ DiscussionBoard.errors[11844605].message }}
  • Profile picture of the author avahadid0
    Yes testing period is long enough
    {{ DiscussionBoard.errors[11844609].message }}
  • That depends on a number of factors. It depends on what you're testing. What your outcomes that you want, it depends on budget etc. What exactly are you testing?
    {{ DiscussionBoard.errors[11844670].message }}
Avatar of Unregistered

Trending Topics