Split Testing May Be Harder Than I Thought!

[shanecabob]

Hey,

So I thought I knew what I was doing when it came to split testing google ads against eachother but I just thought of something...
After you have collected the 30 clicks and are about to delete the loser. Should you view the stats under "all time" "last 7 days" or the "today" and "Yesterday tabs". When I tried both I noticed the CTRs are different depending on what set of data you are looking at.
for example:
Here are my stats when viewing "last 7 days" tab

AD 1
Writing Tutors
Improve Writing Skills
Tutors Are Available 24/7
www.tutor.com/

947 Impr.
15 clicks
1.58% CTR
AD2
Need A Writing Tutor?
Improve Writing Skills
Tutors Are Available 24/7
www.tutor.com/

595 Impr.
15 Clicks
2.52% CTR

Common sense would tell you to delete the top ad.
But when I view it using the "all time" tab, I find that Ad1 has a much better CTR than Ad2. The complete opposite stats!

Which Ad is the real winner?

How do I know which tab to view in order to get accurate data
I'm sure the answer is complete common sense but for some reason I'm not sure what to do. Anybody know?

Shane

[dburk]

Hi Shane,

The larger you dataset the higher your accuracy is. Though you should definitely keep in mind that there are trends. What worked well 6 Months ago may not work as well today. When looking at such a small number of actions (only 15) it can be highly inaccurate.

Another important thing to consider is that's it's better to receive a lower CTR if your conversion rate is higher. What you should be looking at is your cost per conversion.

If your current ads are running very close to the same CTR then you can simply ad more ad variations until you find one that stands out. Again, cost per conversion should be your measuring stick. An effective ad will filter out clicks from visitors that are unlikely to convert.

Now there are many other things to consider, like seasonal trends and current affairs. There are varying levels of competition and major changes in PPC program policies.

Yes, it can be much more complicated than it first appears. Just document everything and consider all factors when making decisions.

[thadroe]

I'd always go for the longer time period for more statistical stability. You've only got 15 clicks on the most recent results, so it may not really be an accurate measure.

But...say you're wondering if the trend might continue...

When you have some conflicting results like that, you might have an external factor like a new competitor stepping in which is affecting your most recent results. Maybe they'll stick around, maybe they won't.

If that or any other variable isn't something you've noticed or can easily account for, If it were my own split test I'd do this:

Continue to split test, but with the long term winning ad at say a 66.6% rotation and the short term winning ad at only a 33.3% rotation.

One way you can force Adwords to do that is by creating an extra copy of your long term winning ad. Adwords doesn't allow identical ad copies, so just change something extremely minor, like a period, comma, etc. Then leave in your short term winner...that will drop it to only a 33% rotation. Doesn't have to necessarily be a 33% rotation...you could create more copies of the winner and do 25%, 20%, etc...whatever you prefer.

That will allow you to take more advantage of the long term winner and keep gathering data on the short term winner, but at a reduced level...and you'll clear up any doubts about the short term trend by seeing if it holds up.

Thad

[shanecabob]

Thanks for the help guys

Shane