9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

Google Unable to Crawl My Website

by tarmizikun
8 replies
  • SEO
    • |
A Google Webmaster error, each time I tried to add sitemap, the will be an error:

"URL restricted by robots.txt"

However the robot never restricted to visit my site. I try the other tool "Fetch by Google". Same error, "Denied by robots.txt

My robots.txt is http://chamoisee.com/robots.txt

The new website utilizes Wordpress 3.3.2

Any help or advice from fellow warrior is highly appreciated.
#crawl #google #google webmaster error #unable #website
  • Profile picture of the author Nile Anderson
    Hi tarmizikun

    There are many pages which you don't want to crawl. Remove old robots file and add new robots file. I think,something should be wrong with your current robot file.Do this and problem still persist then let me know.
    Signature
    SEO Services India |SEO Outsource Online | Web Design Company Orange County | Hudson Shuffleboard
    {{ DiscussionBoard.errors[6325002].message }}
    • Profile picture of the author tarmizikun
      Originally Posted by Nile Anderson View Post

      Hi tarmizikun

      There are many pages which you don't want to crawl. Remove old robots file and add new robots file. I think,something should be wrong with your current robot file.Do this and problem still persist then let me know.
      Hi Nile,

      I did re-create the robots.txt
      The problem still exist in Webmasters
      Honestly I am not sure why this problem occurs as I dont have similar error with other sites.
      Signature
      Malaysia Hotel
      {{ DiscussionBoard.errors[6325574].message }}
  • Profile picture of the author phil.wheatley
    There is a plugin that is good for this, I think it's just called robots. Also, check at the top of your control panel in WP that you haven't got the site set to private.

    Phil
    Signature



    It's still not working for you??? Need direction?...
    ---->>>> BrainDirection.com <<<<----
    {{ DiscussionBoard.errors[6325497].message }}
    • Profile picture of the author tarmizikun
      Originally Posted by phil.wheatley View Post

      There is a plugin that is good for this, I think it's just called robots. Also, check at the top of your control panel in WP that you haven't got the site set to private.

      Phil
      I used the PC Robots txt plugin before to edit the robots txt file.
      In the privacy setting, already ticked "Allow search engines to index this site."

      The problem still exisit in webmasters :-(
      Signature
      Malaysia Hotel
      {{ DiscussionBoard.errors[6325601].message }}
  • Profile picture of the author Chris Silvey
    Originally Posted by tarmizikun View Post

    A Google Webmaster error, each time I tried to add sitemap, the will be an error:

    "URL restricted by robots.txt"

    However the robot never restricted to visit my site. I try the other tool "Fetch by Google". Same error, "Denied by robots.txt

    My robots.txt is http://chamoisee.com/robots.txt

    The new website utilizes Wordpress 3.3.2

    Any help or advice from fellow warrior is highly appreciated.




    Ummm -
    User-agent: Googlebot Disallow:
    Signature
    WP Animate - Increases Conversions & Clicks!
    Create Amazing CSS3 Animations in just a few Clicks - New!
    WPHeadline.net - Create Blazing Headlines in just a few clicks. Updated to WordPress 4.1.1
    {{ DiscussionBoard.errors[6325519].message }}
  • Profile picture of the author tarmizikun
    I decided to move the domain to other server.

    This thread is CLOSED
    Signature
    Malaysia Hotel
    {{ DiscussionBoard.errors[6325877].message }}
    • Profile picture of the author Myles Sinclair
      Originally Posted by tarmizikun View Post

      I decided to move the domain to other server.

      This thread is CLOSED
      The server is not the problem. It seems you have your robots.txt file set to disallow the sitemap!

      As you have nothing else referenced in your Robots file, just delete it and Google will then crawl your sitemap.
      {{ DiscussionBoard.errors[6325908].message }}

Trending Topics