3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

Robots.txt - Why?

by cleptoiii
2 replies
  • SEO
    • |
Hi, two days ago i remove my robots.txt but, even today, google returns me a lot of errors. I show you the image:


How is possible? Now my file robots.txt return 404. Please Help.
#error #google #robotstxt
  • Profile picture of the author jinx1221
    Why did you remove your robots.txt? Just put it back You probably have defined in Webmaster tools that you have one. If you need to just put the file back but leave it blank, or remove from Webmaster tools that you have a robots.txt
    Signature

    The Ultimate Private Network Management,
    Visualization and Automation Tool



    {{ DiscussionBoard.errors[8136014].message }}
  • Profile picture of the author paulgl
    You don't need a robots.txt, but....
    the googlebot will assume it's there and look first for it.
    If it's not found, it will return an "error." It's
    not really an error. It will then crawl everything
    else....(keep reading)

    However, because the robots.txt if the first thing it
    looks for, if it actually cannot find anything else, it
    will also return the robots.txt error, and people will
    assume it's ONLY reporting the robots.txt error and
    not realizing that nothing else can be found. One reason
    to have a robots.txt, even it's blank. If you have a robots.txt
    and google says it can't find it, then you know something is
    wrong at the server level.

    What's funny, is that removing the file gives you one more error.

    But if google says it can't find it, AND, does not crawl anything
    else, it's a server problem as I stated above. Something else
    is blocking the googlebot.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[8136018].message }}

Trending Topics