robots.txt errors

by 1 replies
1
I like to keep my error log down to help when there is a screw up. I always thought that this in the head was enough for the robots.

<meta name="robots" content="index, follow," />

But I keep getting them looking for public/robots.txt and creating an error line. (snarl)

Should I also include a text file in the public folder for them? I guess the content would be

user-agent: *
disallow:

I also hate clogging up my public folder so adding a file for 2 db seems a bit tedious too. (groan) (db is geek speak for dozen bytes)

Also while I was googling this I ran into quite a few comments that there is no point in having the meta tag I quoted above unless you want to disallow. The default behavior for robots is to follow. Guess i should delete them all?

Anybody have any comments about any of this and happen to feel a little chatty?
#website design #errors #robotstxt
  • Just put in a robots.txt file in your root web directory, and then make sure your website is in Google Webmasters and that it validates your robots.txt.

    Also be aware that newer versions of Wordpress have a virtual robots.txt

    Use the meta to dissallow as you have said.

    Aaron
Join the discussion

Next Topics on Trending Feed

  • 1

    robots.txt errors

    I like to keep my error log down to help when there is a screw up. I always thought that this in the head was enough for the robots. <meta name="robots" content="index, follow," />