6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

URL restricted by robots.txt in Webmaster Tools

by Isaan
3 replies
Hello,

I have several websites, but this is the first time I get the following error in Google Webmaster Tools:

URL restricted by robots.txt

I have the "WP Robots Txt" plugin by Christopher Davis installed.

Now I see that it displays a Robots txt Content box in the Privacy Settings of Wordpress.

This is the content that is shown in that box:

---------------------------
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
---------------------------

When I visit the Robots txt file in my browser, it shows the following content:

--------------------------------------------------------------
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Sitemap: http://www.thepowersawsexpert.com/sitemap.xml.gz
--------------------------------------------------------------

Does anybody know what is going wrong here and how I can fix this problem?

Your help is very welcome.


Thanks,

Isaan
#restricted #robotstxt #tools #url #webmaster
  • Profile picture of the author OldLodgeSkins
    Hi,

    To me, nothing's wrong and you don't have any problem...
    The robots aren't allowed to index your admin area. And then? They have nothing to do there anyway. What you saw isn't an error, it's just an information. Google should be indexing the public part of your website just fine.

    Seb.
    Signature
    Do you use Facebook ? Then you can make money just by inviting people to a Facebook group ! It's called the Instant Income System. How cool is that?
    {{ DiscussionBoard.errors[6296345].message }}
  • Profile picture of the author hassan007
    There is nothing wrong, Google just want to keep you notified that you have some links/folders blocked by robots on your site.....

    So no worries at all....

    If you examine the robots.txt file of all major blogs/forums/sites you will find that many of them have blocked some folders in their robots.txt....

    This is done because when you start getting high traffic then suddenly crawler starts indexing your folders and files you don't want to show to other people....

    So just relax, nothing is wrong....

    All the best....
    {{ DiscussionBoard.errors[6296664].message }}
    • Profile picture of the author Isaan
      Hello Hassan and OldLodgeSkins,

      Thank you for taking the time to get me out of that panic.

      I will wait for the first crawling results in the Webmaster Tools.

      Normally I get my sites indexed within 24 hours after pinging the main url in BulkPing.com.
      Now, my site is still not indexed after 24 hours, that's the reason I didn't feel comfortable about the error message.

      Thanks again and I will be patient,


      Isaan
      {{ DiscussionBoard.errors[6296835].message }}

Trending Topics