8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

Help with Robots.txt errors

by tac88
7 replies
Hey fella warriors; I am having a error in my robots.text file this is the error :

"You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file."

Could someone help me to fix this please ..
Thanks
#errors #robotstxt
  • Profile picture of the author David V
    Who is telling you this error?
    {{ DiscussionBoard.errors[7645615].message }}
  • Profile picture of the author dennismw
    I checked my site using my google webmaster tools account and this is how my robots.txt is built. It works fine.

    User-agent: *
    Allow: /

    This info seemed informative. The Web Robots Pages
    {{ DiscussionBoard.errors[7645616].message }}
  • Profile picture of the author tac88
    I used this checker New Robots.txt Syntax Checker: a validator for robots.txt files !
    Signature
    A Simple Formula For Online Success
    This is what's putting money in my pocket Right Now!

    {{ DiscussionBoard.errors[7645716].message }}
  • Profile picture of the author tac88
    So how do I input the disallow command?
    Signature
    A Simple Formula For Online Success
    This is what's putting money in my pocket Right Now!

    {{ DiscussionBoard.errors[7645723].message }}
  • Profile picture of the author David V
    eglassman88 is right, crawling happens by default.

    This is a robots.txt entry that blocks the whole site: (Disallow everything)
    User-agent: *
    Disallow: /


    This is a robots.txt entry that allows full access to all robots: (Disallow nothing)
    User-agent: *
    Disallow:

    The complete blank after the "Disallow:" indicates it's disallowing nothing.
    You should really take advantage of Google webmaster tools, they can really help you with crawling errors, what should be blocked, and obviously much more then just the site access issues.

    FYI:
    Google and Bing accept 2 specific regular expression characters for pattern exclusion (* and $).

    Here's the minimum I start with in my WP sites:
    User-agent: *
    Disallow: /wp-* (all directories that start with wp-)
    Disallow: *.php (all directories that end with .php RE:wp-config.php, wp-cron.php, et..)
    Disallow: */feed* (don't need feed listed)
    Disallow: */trackback* (don't need trackbacks listed)
    Disallow: /cgi-bin/
    {{ DiscussionBoard.errors[7645852].message }}

Trending Topics