How to fix crawler error ?

by 9 replies
10
My website has 119 server error (code : 500) and not found 201 (error code:404). What should i do. should i collect all of them and disallow them on robots.txt file or any other way to fix them? Please let me know someone.
#search engine optimization #crawler #error #fix
  • I believe the spider is crawling your site but found some internal links that does not exist. You may want to find out where these bad links are and make them good, else it will be frustrating for your visitors as well when they click on the bad links.

    try this:
    Free Broken Link Checker / Online URL Validator - finds bad / dead weblinks
  • Robertarticle,

    You would be better off redirecting those pages to pages that are working. You can view detailed reports in webmaster tools about which links are causing issues and you will be able to see exactly where the links are. You can then either fix them or redirect them to working pages.

    Here is info on how to deal with them as well,

    404 (Not found) - Webmaster Tools Help

    Best,

    Shawn
    • [1] reply
    • There's no reason to redirect. Just get a custom 404 error.

      It does not matter, unless those urls are ones that are
      SUPPOSED TO BE THERE!

      It's a common occurrence, and no reason for anyone to
      panic.

      If you actually coded the site yourself, and did not use
      some lame CMS, like wordpress, there would be no
      problem anyway.

      It's time for people who want to be webmasters, taking
      the bull by the horns, and actually becoming webmasters!

      Learn something!

      Paul
  • If i block all the crawler error on robots.txt file. Is it a good practice?
    • [1] reply
    • Hmm that's not a problem.. you can do that.. Or just remove those broken links, this will be a good one..
  • Simple. Just select all and click mark as fixed.
    • [1] reply
    • Haa don't you know it shows the error again.. This error will be seen till its fixed.
      • [1] reply
  • What you have done is not for good.. You should understand that first.. The alert sent by WMT is not the matter, it has to be fixed else you suffer.
Join the discussion

Next Topics on Trending Feed

  • 10

    How to fix crawler error ?

    My website has 119 server error (code : 500) and not found 201 (error code:404). What should i do. should i collect all of them and disallow them on robots.txt file or any other way to fix them? Please let me know someone.