7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Refreshing robotx.txt?

by RonnyRaygun
6 replies
  • SEO
    • |
So I spent the last week or so building my wordpress site, during which I used the privacy settings to block the search engines from indexing my site.

I turned off the privacy settings and tried to see if google will index my site. I looked at the google webmasters page area and it keeps saying that googlebot is blocked by robots.txt I changed the robots.txt manually and googlebot page checker is still showing that robots.txt is blocking googlebot.

What am I doing wrong?

Please advise
#refreshing #robotxtxt
  • Profile picture of the author yukon
    Banned
    Can you post the robots.txt file here?

    Hard to say without knowing what's inside the file.
    {{ DiscussionBoard.errors[4141639].message }}
  • Profile picture of the author RonnyRaygun
    This is the new robots.txt file (links edited because I have so few posts):

    Code:
    User-agent: * 
Allow: / 

 Sitemap: [domainname]/sitemap.xml.gz
    This is what I see in the google webmaster area

    Code:
    User-agent: *
Disallow: /

Sitemap: [domainname]/sitemap.xml.gz
    Signature

    {{ DiscussionBoard.errors[4141704].message }}
  • Profile picture of the author yukon
    Banned
    Here is what I use for my robots.txt, looks like your running the same Google Sitemap (sitemap.xml.gz) Wordpress plugin that I run.

    Sitemap: http:// [domain] /sitemap.xml.gz
    User-agent: *
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-login.php


    1) First thing you need to do is a browser search for your sitemap, when you do this search make sure to leave off the very end of the sitemap url ( .gz ). If you can load your sitemap.xml file in your browser than Google will also be able to read the xml file.

    Example URL: http:// [domain] /sitemap.xml

    All the .gz does is zip the xml file, makes it faster to download the xml file.

    2) While your in the GWT (Google Webmaster Tools) Admin. go ahead & delete the old GWT sitemap that you already submitted. Do not try to Resubmit the sitemap. You need to submit as a new sitemap, just make sure the old sitemap is deleted inside GWT first.

    3) After you Submit (NOT Resubmit), refresh your browser window a couple of times. The new sitemap should be accepted.
    {{ DiscussionBoard.errors[4141814].message }}
  • Profile picture of the author yukon
    Banned
    BTW, Google shows 100% of my Wordpress pages indexed inside GWT.
    {{ DiscussionBoard.errors[4141818].message }}
  • Profile picture of the author RonnyRaygun
    Thanks for your help yukon,

    I just noticed something in GWT, it says the last time the robots.txt file was downloaded was 18 hrs ago. How do I get GWT to download the new robots.txt? I think this is the cause of my problems.
    Signature

    {{ DiscussionBoard.errors[4141997].message }}
    • Profile picture of the author yukon
      Banned
      Originally Posted by RonnyRaygun View Post

      Thanks for your help yukon,

      I just noticed something in GWT, it says the last time the robots.txt file was downloaded was 18 hrs ago.
      I wouldn't worry about that for now.

      The important thing is that the new sitemap.xml file is valid & GWT shows it as ok (green checkmark) in GWT > Site configuration > Sitemap



      Originally Posted by RonnyRaygun View Post

      How do I get GWT to download the new robots.txt? I think this is the cause of my problems.
      Follow the steps I posted above, delete the old sitemap inside GWT, then Submit the new sitemap (Don't use the Resubmit option).
      {{ DiscussionBoard.errors[4142096].message }}

Trending Topics