How do you split-test?
 

I'm testing response rates to my Craigslist ad.
Here are some major elements I can test:
a) headline
b) location
c) picture/no picture

I have two different accounts to post ads. So my strategy is posting the same ad (hence same headlines and picture/no picture) in city A and city B one week, then change the headline for weeks 2 and 3. With the headline winner, repeat with testing c. Then post the same ad in different cities.

If you were in my shoes, what would you change about the testing process? Does it sound like the testing methods are on the right track?

As an aside, when do you tell whether the mental costs of split-testing outweighs the resulting benefits?

Thank you in advance for your feedback!

Re: How do you split-test?
 

Hi stanigator,

What you are proposing to do is called a serial test, not a split test. A serial test is not quite as accurate as a split test and you generally need to let your test run for a longer period of time to get enough data for a valid conclusion.

For best results make sure that you are changing just one element at a time and that you are making a significant change. For example, when testing headlines, do not make a minor insignificant change to the headline, make it radically different so that you are more likely to see at least a 20% difference in performance, otherwise the test results will hold no significance.

Be sure to setup your tests with a specific purpose in mind for what you are to learn as a result of the test. This is commonly referred to as a hypothesis. Analyse the results, of testing your hypothesis, to form a theory as to why it worked the way it did. Next, devise tests that will disprove your the theory if it isn't completely valid. If your theory holds true during your validation tests, you have a winning insight that can be applied to your entire marketing strategy.

I suggest you start by identify various value propositions and test those against each other first, so that when you find a winner it will be clear that it is the value proposition rather than clever wording that people are responding to. Insights, into value propositions that work, have the broadest potential application for conversion rate optimisation, IMHO.

As long as your testing is showing significant benefits you should continue. In marketing you should never stop testing until you have no more competitors and are ready to retire. ;)

Re: How do you split-test?
 

I was suggesting a serial test b/c split-testing isn't practical in my environment, as I wasn't allowed to post multiple ads with the same account. This leaves me to one more question: when do you use split-test vs. serial-test? Is it an accuracy vs. practicality tradeoff?

Re: How do you split-test?
 

Hi stanigator,

Generally, I would recommend split testing over serial testing. The only time you should use serial testing is when it is more practical than split testing, as in the case you described.

Re: How do you split-test?
 

What factors determine the practicality? I may have another situation that says so.

Re: How do you split-test?
 

Anything that prohibits you from using 2 or more alternative page variants at the same time would make split testing impractical.

If you can use javascript code snippets or run simultaneous variants of a page then you can, and should, split test.