I have disallowed all the bots to crawl my website because it is under construction.What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.The website has some important company data they don't want anyone to access, that's why i can't allow all.
ErrorThis URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.
#allowing #analytics #crawl #online #seo #tools #website
My Must Haves List: Wordpress, GoStats (website analytics), Google Trends
If you were disappointed in your results today, lower your standards tomorrow.