Am I Blocked Yahoo's Crawler? NING/1.0
I was wondering if I'm blocking Yahoo's crawler. When I look in Google, I have over 250,000 pages indexed while in Yahoo, I have only 300 or so.
I blocked a bot with user agent NING/1.0. When running IP address queries on them, I noticed that some of their IPs come from Amazon and others come from Yahoo. I know for a fact that they're just hitting up my 301 redirects from my own shortened URL host (100% of them). You can see that they've hit my page NUMEROUS times this month.
Any conclusive evidence of NING being Yahoo's Crawler?
1 67.195.111.163
1 72.30.161.227
1 b3091305.crawl.yahoo.net
1 llf531090.crawl.yahoo.net
161 212.124.109.166
2068 46.51.132.92
65124 67.195.111.230
126726 72.30.142.174
I blocked a bot with user agent NING/1.0. When running IP address queries on them, I noticed that some of their IPs come from Amazon and others come from Yahoo. I know for a fact that they're just hitting up my 301 redirects from my own shortened URL host (100% of them). You can see that they've hit my page NUMEROUS times this month.
Any conclusive evidence of NING being Yahoo's Crawler?
1 67.195.111.163
1 72.30.161.227
1 b3091305.crawl.yahoo.net
1 llf531090.crawl.yahoo.net
161 212.124.109.166
2068 46.51.132.92
65124 67.195.111.230
126726 72.30.142.174