Google Web Crawl Problem

by Unexpected Error Posted: 17 years ago 10 replies

Hi Guys,

On checking Diagnostics for my website on google's Webmaster tools page, it says "We had problems crawling the pages listed here". It then gives this page: http://MySite.com/index.html and says 100 pages are linking to it.

Well, firstly, the sites linking to it give this url: http://MySite.com - NOT http://MySite.com/index.html. I should know, I've built those links!

It's true I haven't got a page called http://MySite.com/index.html - but as I've said, the links DON'T point to that url anyway!

Can someone be good enough to help me out. This is very confusing!

Thanks,

Simon

#search engine optimization #crawl #google #problem #web

angilina 17 years ago

You can simply do a 301 redirect from http://MySite.com/index.html to http://MySite.com.

I think that can solve the problem?
- [1] reply
- Unexpected Error 17 years ago
  
  I don't think so, Angilina. Google wants to actually crawl http://MySite.com/index.html but can't - since that page doesn't exist! So a re-direct wouldn't solve the problem that google says it has.
  
  But the other thing, of course, is WHY it wants to crawl http://MySite.com/index.html from links that point to http://MySite.com!!!
  
  [1] reply
tamtu 17 years ago

By using Google's new canonical link attribute you can give specific direction as to which link to crawl. Check that out.
- [1] reply
- Unexpected Error 17 years ago
  
  Ok, thanks. I'll try it.

Join the discussion

Next Topics on Trending Feed

12

Google Web Crawl Problem

Unexpected Error • 17 years ago • 10 replies

Hi Guys, On checking Diagnostics for my website on google's Webmaster tools page, it says "We had problems crawling the pages listed here". It then gives this page: http://MySite.com/index.html and says 100 pages are linking to it.