Split Testing May Be Harder Than I Thought!

Hi Shane,

The larger you dataset the higher your accuracy is. Though you should definitely keep in mind that there are trends. What worked well 6 Months ago may not work as well today. When looking at such a small number of actions (only 15) it can be highly inaccurate.

Another important thing to consider is that's it's better to receive a lower CTR if your conversion rate is higher. What you should be looking at is your cost per conversion.

If your current ads are running very close to the same CTR then you can simply ad more ad variations until you find one that stands out. Again, cost per conversion should be your measuring stick. An effective ad will filter out clicks from visitors that are unlikely to convert.

Now there are many other things to consider, like seasonal trends and current affairs. There are varying levels of competition and major changes in PPC program policies.

Yes, it can be much more complicated than it first appears. Just document everything and consider all factors when making decisions.

I'd always go for the longer time period for more statistical stability. You've only got 15 clicks on the most recent results, so it may not really be an accurate measure.

But...say you're wondering if the trend might continue...

When you have some conflicting results like that, you might have an external factor like a new competitor stepping in which is affecting your most recent results. Maybe they'll stick around, maybe they won't.

If that or any other variable isn't something you've noticed or can easily account for, If it were my own split test I'd do this:

Continue to split test, but with the long term winning ad at say a 66.6% rotation and the short term winning ad at only a 33.3% rotation.

One way you can force Adwords to do that is by creating an extra copy of your long term winning ad. Adwords doesn't allow identical ad copies, so just change something extremely minor, like a period, comma, etc. Then leave in your short term winner...that will drop it to only a 33% rotation. Doesn't have to necessarily be a 33% rotation...you could create more copies of the winner and do 25%, 20%, etc...whatever you prefer.

That will allow you to take more advantage of the long term winner and keep gathering data on the short term winner, but at a reduced level...and you'll clear up any doubts about the short term trend by seeing if it holds up.

Thad