Don't jump to conclusions when testing conversion

We often hear that it takes around 1000 unique visitors to accurately test conversion. But is this all you need? What about minimum time exposure?

I often handle my own phone orders (a great way to talk to customers, which so many of us rarely or never do) and one thing I've learned is that many people don't buy on the first visit to a website. I've had customers tell me they've been hanging around the site for a couple weeks or more.

So how does this apply to conversion testing? Well, if you're comparing minor elements it may not have much impact. But if you're testing major differences like copy, layout or appearance it could play a big role.

Depending on your traffic you could have 1000 uniques in a day or even hours. If you commit to a test result in that short length of time, even with 1000's of visitors, you still have no idea about the long term effect on visitor behaviour of the pages you tested. You may have prematurely pulled a winner.

I haven't come up with an answer to this that covers all the bases. But it definitely bears thinking about.