Using different %'s of traffic
 

I've been running a few tests using varying % levels of traffic and each time I seem to get slightly different results. The first two times I ran it, I used 10% of total traffic; the first test got no significance, the second time got a decrease in conversion rate with significance, and the third time I ran it with 100% of traffic and got an increase in conversion rate with significance.

All 3 tests sway in varying directions. Has anyone encountered this experience? And how do you make a call on something like this?

Re: Using different %'s of traffic
 

I guess you need to redo the tests with the same percentages of traffic to see if you will get the same conversation rates again.

Re: Using different %'s of traffic
 

Much more than percentage of traffic matters homogeneity of traffic. Have you used the same traffic sources during all of the three tests?

Re: Using different %'s of traffic
 

What are the sample sizes here?

When you say 10% of total traffic, did this total 1000 unique visits?

You really need to run 1000 visits to an A/B test before having data you can rely on, long term... And, depending on your audience size, your niche and a few other factors, you could potentially bump into the law of diminishing returns.