Bing Webmaster Center blog discusses crawling and robots.txt file:
|
...As much as robots.txt is a reference tool to control BingBot's behavior, it is also a double-edged sword that may be interpreted in a way that disallows (or allows) much more than you thought initially. In this column, we will go through the most common robots.txt directives supported by Bing, highlighting a few of their pitfalls, as seen in real-life feedback over the past few months....
|
To crawl or not to crawl, that is BingBot's question - Bing Webmaster Center Blog
