I've developed ajax driven survey questionnaire for my users. Possible outcomes for users are in the hundreds. Each outcome is slightly different. If I do a spider crawl check on the page, I noticed spiders dig through the survey and see all the survey content . Since a ton of this information is the same it's going to look like spam. How can I block bots from crawling the survey? I've blocked via robots.txt but it still doesn't matter. I thought since this is my database the spider needs password access.