5 {{ upvoteCount | shortNum }}
5 {{ upvoteCount | shortNum }}

Need help with my robots.txt file please...

by Si_P
2 replies
Hi,

Like many of my other money sites, they are hosted with Hostagator and in the form of Wordpress blog. Never have I encountered this problem before as i have never had to set up a robots.txt file manually.

For some reason when I try to submit my sitemap to google, i get a crawl error that my robots.txt file has the txt:

User-agent: *
Disallow: /

which because of the '/' after 'Disallow:', my complete site cannot be crawled.

I never set this up to be like that and i havent a clue how to change it. I decided to manually create a new robots.txt file with:

User-agent: *
Disallow:

so that my site should be crawled. I uploaded it to the '/public_html/mydomain dot com' directory about 10 hours ago, but it now lists all the page addresses which cannot be accessed and shows the following after testing the file in my webmaster tools section.

Url Googlebot Googlebot-Mobile mydomain dot com/ Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions
Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions

How on earth do i fix this as i need my site indexed asap by the Googlebot?

Thanks,
Si
#file #robotstxt
  • Profile picture of the author Bruce Hearder
    You can use both formats :

    User-agent: *
    Disallow:

    or

    User-agent: *
    Allow: /

    Sometimes it takes BigG a whiole to come back again.
    I would make sure you log into Google Webmasters, and go the domain in question.
    Click on "Site Configuration" , then "Crawler Access".

    At the bottom, of that page is a "Test" button.

    Click that to make sure that Google can read your robots.txt file.

    Also, a catch for "young players" is that in the latest version of WP (2.8.4) its privacy default is set so that Search Engines can't read your site.

    Go to Settings->Privacy and check which option is checked.

    The default is :

    "I would like to block search engines, but allow normal visitors"

    It should be changed to "I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers"

    Then press the "Save Changes Button"

    I hope this helps

    Bruce
    {{ DiscussionBoard.errors[1264028].message }}
    • Profile picture of the author Si_P
      Hi Bruce,

      Thanks for the info. I had the new version of Wordpress on the site so had to change the privacy settings.

      It is allowed now.
      {{ DiscussionBoard.errors[1266814].message }}

Trending Topics