29 {{ upvoteCount | shortNum }}
29 {{ upvoteCount | shortNum }}

What do you mean test?

by DavidLasry
25 replies
I keep hearing people in the forum say that the key is to test, test, test but I never actually understood that. For instance, I've been trying to study POF as a traffic source for couple of months and when it comes to test more, I find myself pretty lost.

When I start campaigns, I usually make a batch of creatives to test, something like 80 creatives and then test them for a whole week. After a week, I get rid of the losers and keep all the ads with 0.1xx CTR or higher.

At this point I feel kind of lost because everybody keep telling me to test new creatives but I don't know how and in which order. Also, sometimes, the ads that had 0.1xx CTR or higher now has very poor CTR and I don't know how to manage it.

What I'm trying to say is that I know how to start testing but I get stuck everytime when it comes to test another creatives or to move on to the next level.

If someone could point me to the right direction or to give me some tips I would really appreaciate it!

Sincerely,
David.
  • Profile picture of the author PaulBaker
    Originally Posted by DavidLasry View Post

    When I start campaigns, I usually make a batch of creatives to test, something like 80 creatives and then test them for a whole week. After a week, I get rid of the losers and keep all the ads with 0.1xx CTR or higher.
    Hey David,

    Testing 80 ads over a week may not be the best use of your money. Try testing something like 10 at a time.

    I like to test an ad for a minimum 2x the offer payout ($5 offer = $10 spent per creative). So testing 80 ads would cost you $800 to get any real indication of what is or isn't working.

    Don't get too caught up on CTR. High CTR ads aren't always the most profitable nor are they always the best quality leads for the advertiser/merchant. Conversions/profit is what's important. After running an ad for 2x offer payout, did you make money or loose money? Keep the goodies and replace the losers.

    Personally I find it more important to test landers and offers first.

    Offers I judge by EPC and Landers I judge by CVR. Before I make any descisions on these I ideally want multiple conversions and a few hundred clicks.

    Hope it helps as a new starting point. Great to see you persevering
    Signature

    Put Your Money Where Your Mouth is: CLICK HERE

    {{ DiscussionBoard.errors[8857850].message }}
    • Profile picture of the author DavidLasry
      Originally Posted by PaulBaker View Post

      Hey David,

      Testing 80 ads over a week may not be the best use of your money. Try testing something like 10 at a time.

      I like to test an ad for a minimum 2x the offer payout ($5 offer = $10 spent per creative). So testing 80 ads would cost you $800 to get any real indication of what is or isn't working.

      Don't get too caught up on CTR. High CTR ads aren't always the most profitable nor are they always the best quality leads for the advertiser/merchant. Conversions/profit is what's important. After running an ad for 2x offer payout, did you make money or loose money? Keep the goodies and replace the losers.

      Personally I find it more important to test landers and offers first.

      Offers I judge by EPC and Landers I judge by CVR. Before I make any descisions on these I ideally want multiple conversions and a few hundred clicks.

      Hope it helps as a new starting point. Great to see you persevering
      I understand what you're saying but how can I test 10 ads at a time when the time could be a factor here? What I mean is that if I test 10 creatives and next week I would test another 10, maybe the time would interfere my testing and result in bad data.

      Also, I'm looking for tips about testing in small budget($200).
      Signature

      Sincerely,
      David.

      {{ DiscussionBoard.errors[8858984].message }}
      • Profile picture of the author PaulBaker
        Originally Posted by DavidLasry View Post

        I understand what you're saying but how can I test 10 ads at a time when the time could be a factor here? What I mean is that if I test 10 creatives and next week I would test another 10, maybe the time would interfere my testing and result in bad data.

        Also, I'm looking for tips about testing in small budget($200).
        Don't over complicate things. You're looking for ads that are profitable, that's all.

        You can't adequately test 80 ads for $200. Start small, make some money and re-invest it.

        I'm not sure what else to tell you.
        Signature

        Put Your Money Where Your Mouth is: CLICK HERE

        {{ DiscussionBoard.errors[8859273].message }}
  • Profile picture of the author domainarama
    99.9999% of what you read here on WF about testing is sheer nonsense. As someone who has taught the stuff in graduate school I tell you any real, genuine test comparing one A to another B or C, D.E...etc costs $$$thousands and is very difficult. Maybe you can use one of the AB testing programs which are sold here and get results which indicate that this A is better than that B. But I'll guarantee you that you can repeat the AB testing next week or next month and get the exact opposite results.

    Your intuition is probably better than any testing you can afford. Just make sure you get up on the right side of the bed.
    {{ DiscussionBoard.errors[8859305].message }}
    • Profile picture of the author vask
      Originally Posted by domainarama View Post

      99.9999% of what you read here on WF about testing is sheer nonsense. As someone who has taught the stuff in graduate school I tell you any real, genuine test comparing one A to another B or C, D.E...etc costs $$ and is very difficult. Maybe you can use one of the AB testing programs which are sold here and get results which indicate that this A is better than that B. But I'll guarantee you that you can repeat the AB testing next week or next month and get the exact opposite results.
      A lot of people here don't have the luxury of testing that extensively. Companies can pour in thousands of dollars into A/B testing and can come up with a solid 90%+ confidence level.

      Although testing on a smaller scale isn't accurate, it is better than not testing at all.

      Your intuition is probably better than any testing you can afford. Just make sure you get up on the right side of the bed.
      I 100% agree with this.
      {{ DiscussionBoard.errors[8862984].message }}
      • Profile picture of the author domainarama
        Originally Posted by vask View Post

        Although testing on a smaller scale isn't accurate, it is better than not testing at all.
        WRONG!!! Maybe you would like to think that. But you'd be wrong, wrong, wrong.

        If you do bad, fake, deluded, cheapo testing today and find that A is 20% better than B, I GUARANTEE that if you do the same testing next week/month/time period, you are extremely likely to find that B is 20% better than A (it's sometimes called 'regression to the mean').

        There is a whole world of professional testing tools and expertise. The way most A/B testing is done on WF sites is like the Little Leagues compared to the Major Leagues. Any resemblance is purely coincidental.

        Would you say that a little brain surgery done by a guy with a pen knife and some bandaids is better than not going to a certified brain surgeon?
        {{ DiscussionBoard.errors[8863195].message }}
        • Profile picture of the author vask
          Originally Posted by domainarama View Post

          WRONG!!! Maybe you would like to think that. But you'd be wrong, wrong, wrong.

          If you do bad, fake, deluded, cheapo testing today and find that A is 20% better than B, I GUARANTEE that if you do the same testing next week/month/time period, you are extremely likely to find that B is 20% better than A (it's sometimes called 'regression to the mean').

          There is a whole world of professional testing tools and expertise. The way most A/B testing is done on WF sites is like the Little Leagues compared to the Major Leagues. Any resemblance is purely coincidental.

          Would you say that a little brain surgery done by a guy with a pen knife and some bandaids is better than not going to a certified brain surgeon?
          Well shit, then all the money I've made must've been purely coincidental. In that case, I must be pretty lucky. Hope my luck doesn't run out anytime soon. :p
          {{ DiscussionBoard.errors[8863255].message }}
  • Profile picture of the author PPC-Coach
    Yeah you might be over testing.

    Create 2 ads, split test those for a 3 day period. Or longer if you want more accurate results. Then take the winner and get rid of the loser. Run the winning ad 80% of the time and create another test ad to run 20% of the time. Compare again.

    Don't focus on CTR though. Focus on ROI. That's what matters!

    Signature
    {{ DiscussionBoard.errors[8859307].message }}
    • Profile picture of the author DavidLasry
      Originally Posted by PPC-Coach View Post

      Yeah you might be over testing.

      Create 2 ads, split test those for a 3 day period. Or longer if you want more accurate results. Then take the winner and get rid of the loser. Run the winning ad 80% of the time and create another test ad to run 20% of the time. Compare again.

      Don't focus on CTR though. Focus on ROI. That's what matters!
      So basically you're saying I should always keep testing ads and find winners. Is that what you're saying? If that's so then at what poing will I scale up this business?

      Thanks for your help!
      Signature

      Sincerely,
      David.

      {{ DiscussionBoard.errors[8862875].message }}
    • Profile picture of the author Inferno272
      Originally Posted by PPC-Coach View Post

      Yeah you might be over testing.

      Create 2 ads, split test those for a 3 day period. Or longer if you want more accurate results. Then take the winner and get rid of the loser. Run the winning ad 80% of the time and create another test ad to run 20% of the time. Compare again.

      Don't focus on CTR though. Focus on ROI. That's what matters!
      its amazing how often people LITERALLY split test at 50/50. why would you put a winner on as often as an unknown.
      {{ DiscussionBoard.errors[8868544].message }}
  • Profile picture of the author DavidLasry
    Can someone please explain to me what is A/B testing?
    Also if you can help me out with the process of testing I would really appreaciate it.
    I mean how to test, generally.
    Signature

    Sincerely,
    David.

    {{ DiscussionBoard.errors[8863783].message }}
    • Profile picture of the author Greedy
      Originally Posted by DavidLasry View Post

      Can someone please explain to me what is A/B testing?
      Also if you can help me out with the process of testing I would really appreaciate it.
      I mean how to test, generally.
      A/B testing is when you test 2 different variations or versions of something.

      For example version A and version B.

      Typically you will be testing both versions and the same time to see which preforms better.
      Signature
      {{ DiscussionBoard.errors[8864307].message }}
      • Profile picture of the author sonixkid3000
        IMO, it doesn't really make sense to test by days, so much as impressions. If you don't have a click by 1,000 impression - 1,200 for good measure... shut that creative down. It's not a performer. Your images are fine to have that many but you should also be testing by age group, gender, and any other relevant factors.

        An important piece here is that your campaigns might work for two different age groups or demographics, but your creatives will differ in performance between the two demographics.

        Hope that helps!
        {{ DiscussionBoard.errors[8864415].message }}
        • Profile picture of the author DavidLasry
          Originally Posted by sonixkid3000 View Post

          IMO, it doesn't really make sense to test by days, so much as impressions. If you don't have a click by 1,000 impression - 1,200 for good measure... shut that creative down. It's not a performer. Your images are fine to have that many but you should also be testing by age group, gender, and any other relevant factors.

          An important piece here is that your campaigns might work for two different age groups or demographics, but your creatives will differ in performance between the two demographics.

          Hope that helps!
          Why do you think it's better to test by impressions, rather than by time?
          Don't you think time could be an important factor regarding CTR of ads?
          Signature

          Sincerely,
          David.

          {{ DiscussionBoard.errors[8866524].message }}
          • Profile picture of the author sonixkid3000
            Originally Posted by DavidLasry View Post

            Why do you think it's better to test by impressions, rather than by time?
            Don't you think time could be an important factor regarding CTR of ads?
            Time is a very important factor in the CTR of your ads. But so is money management. That's why if you don't have a click by 1,200 impressions and you keep it going, you're only spending your money - and wasting it. Unless by some freak chance there's a hiccup, that ad isn't going to perform well.

            Think of it like this:

            Why would I spend 3-4 days testing a creative that I know isn't performing well, when I knew that information 8 hours or so after it was approved? It makes no sense to keep it running.

            Testing by impressions is better than testing by days because one creative could have 10,000 impressions in a day and the other could have 2,000 impressions in a day. Regardless, if it's not performing well I wouldn't keep them anywhere past 3,000 impressions. You're just wasting money at that point. Once you're at 1,200 impressions without a click, it should be a red flag for you.
            {{ DiscussionBoard.errors[8871925].message }}
            • Profile picture of the author DavidLasry
              Originally Posted by sonixkid3000 View Post

              Time is a very important factor in the CTR of your ads. But so is money management. That's why if you don't have a click by 1,200 impressions and you keep it going, you're only spending your money - and wasting it. Unless by some freak chance there's a hiccup, that ad isn't going to perform well.

              Think of it like this:

              Why would I spend 3-4 days testing a creative that I know isn't performing well, when I knew that information 8 hours or so after it was approved? It makes no sense to keep it running.

              Testing by impressions is better than testing by days because one creative could have 10,000 impressions in a day and the other could have 2,000 impressions in a day. Regardless, if it's not performing well I wouldn't keep them anywhere past 3,000 impressions. You're just wasting money at that point. Once you're at 1,200 impressions without a click, it should be a red flag for you.
              Well, I guess I'll have to check the efficiency of this and post here what I come up with. Anyway, I hope it's a better indicator about the quality of the ads.
              Signature

              Sincerely,
              David.

              {{ DiscussionBoard.errors[8873087].message }}
    • Profile picture of the author sonixkid3000
      Originally Posted by DavidLasry View Post

      Can someone please explain to me what is A/B testing?
      Also if you can help me out with the process of testing I would really appreaciate it.
      I mean how to test, generally.
      You really have testing down... different creatives, different demos in different campaigns, etc

      Anything past that, try changing one variable at a time and record the results, see if that variable worked for your or not.
      {{ DiscussionBoard.errors[8864417].message }}
  • Profile picture of the author PPC-Coach
    If you have a small budget, you're going to be relying on luck more then anything.

    Signature
    {{ DiscussionBoard.errors[8864588].message }}
    • Profile picture of the author PaulBaker
      Originally Posted by PPC-Coach View Post

      If you have a small budget, you're going to be relying on luck more then anything.
      Luck or good instinct and experience? Experience is often gained by throwing crap at the wall and seeing what sticks.

      from your teespring thread:
      Originally Posted by PPC-Coach

      You can tell pretty quickly what is going to work and what isn't.
      Originally Posted by domainarama View Post

      WRONG!!! Maybe you would like to think that. But you'd be wrong, wrong, wrong.

      If you do bad, fake, deluded, cheapo testing today and find that A is 20% better than B, I GUARANTEE that if you do the same testing next week/month/time period, you are extremely likely to find that B is 20% better than A (it's sometimes called 'regression to the mean').

      There is a whole world of professional testing tools and expertise. The way most A/B testing is done on WF sites is like the Little Leagues compared to the Major Leagues. Any resemblance is purely coincidental.

      Would you say that a little brain surgery done by a guy with a pen knife and some bandaids is better than not going to a certified brain surgeon?
      This is nice theory but we are here to make money.

      Sometimes, perfection and even statistical significance are not the wisest of investments. With the limited life span of most AM campaigns you are often better served making hay while the sun shines. You can eek out 1% here and 3% there or you can take what's working and scale.

      If I have a choice of spending $500 at 100% ROI or 2k at 50% I know what I will choose.

      Originally Posted by DavidLasry View Post

      Also if you can help me out with the process of testing I would really appreaciate it.
      I mean how to test, generally.
      As you can see from this thread, there is not one clear cut answer. You need to weed through all the ideas and opinions and find what makes sense to you and your budget. Then you need to test it
      Signature

      Put Your Money Where Your Mouth is: CLICK HERE

      {{ DiscussionBoard.errors[8864692].message }}
  • Profile picture of the author myleslast
    Don't overwhelm yourself too much at the beginning. It'll only confuse you. Try start slow and increase your campaigns along the way. Also putting yourself in the shoes of your audience and trying to think like them will help a lot. "follow" is a good place to start.
    {{ DiscussionBoard.errors[8866805].message }}
    • Profile picture of the author DavidLasry
      Originally Posted by myleslast View Post

      Don't overwhelm yourself too much at the beginning. It'll only confuse you. Try start slow and increase your campaigns along the way. Also putting yourself in the shoes of your audience and trying to think like them will help a lot. "follow" is a good place to start.
      I know that but what I'm trying to figure out is the main phases of testing. I'm looking for some guidelines that I can follow generally. I'm not looking for a secret formula that will make me rich overnight. I'm just searching general concept that I can follow because on the advanced stages of testing I can find myself pretty lost.

      Do you know what I'm saying?
      Signature

      Sincerely,
      David.

      {{ DiscussionBoard.errors[8867506].message }}
  • Profile picture of the author domainarama
    Superbowl coming up. It's a simple A/B test. If the two teams who meet in the Superbowl also met at some time during the season we know how the Superbowl will turn out! If team A beat team B when they met earlier in the season, we know for sure that team A will beat team B in the Superbowl also. My handy-dandy A/B testing program told me so. A/B testing never lies.

    /sarcasm
    {{ DiscussionBoard.errors[8868459].message }}
    • Profile picture of the author vask
      Originally Posted by domainarama View Post

      Superbowl coming up. It's a simple A/B test. If the two teams who meet in the Superbowl also met at some time during the season we know how the Superbowl will turn out! If team A beat team B when they met earlier in the season, we know for sure that team A will beat team B in the Superbowl also. My handy-dandy A/B testing program told me so. A/B testing never lies.

      /sarcasm
      Dude stop being such a douche. Okay okay, I know. You've "taught this in graduate school" and you're mister big shot and we are all in the "little leagues".

      Let me throw some numbers at you then.

      I netted low 6 figures last year with my affiliate campaigns and many of my campaigns are based off of less than $1000 in testing before I decide to scale it up. And guess what? It's working.

      Low 6 figures may not be a lot to you, but it sure is not just a "coincidence".

      When all you have is a pen and a knife, you just gotta make due.

      Unless you got something to contribute, don't contribute at all. /sarcasm
      {{ DiscussionBoard.errors[8868734].message }}
      • Profile picture of the author DavidLasry
        Originally Posted by vask View Post

        Dude stop being such a douche. Okay okay, I know. You've "taught this in graduate school" and you're mister big shot and we are all in the "little leagues".

        Let me throw some numbers at you then.

        I netted low 6 figures last year with my affiliate campaigns and many of my campaigns are based off of less than $1000 in testing before I decide to scale it up. And guess what? It's working.

        Low 6 figures may not be a lot to you, but it sure is not just a "coincidence".

        When all you have is a pen and a knife, you just gotta make due.

        Unless you got something to contribute, don't contribute at all. /sarcasm
        That's quite an income! I Hope to earn like that soon
        Signature

        Sincerely,
        David.

        {{ DiscussionBoard.errors[8871782].message }}
  • Profile picture of the author Inferno272
    you can live very comfortably on a low 6figure salary.

    hell, you can live pretty well on a mid 5 figures salary if youre frugal.
    {{ DiscussionBoard.errors[8869712].message }}

Trending Topics