i got an error with google bots crawl

by vbskss
3 replies
Hi all,

I have an error that i just can't get ride of after trying alot of things it says that my domain is restricted but i rank my other pages on google and i have checked my robots.txt and it allows the robots to fetch my website after some research i found out that maybe the .htaccess could be the problem this is what i have there.

# -FrontPage-
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*
<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName **********.com
AuthUserFile /home/*****/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/*****/public_html/_vti_pvt/service.grp

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

What is this indexignore .htaccess and all those deny allow can i remove all this.

Ihad also another question one thing that i am not sure of is i have pages like my contact privacy and disclosure policy ranking for my keyword but not just the domain it self should i block them in my robots.txt file or just leave them.

Thanks any help would be greatly appreciated.
#bots #crawl #error #google
  • Profile picture of the author Christian Little
    Copy .htaccess to htaccess.bak, and then go back to .htaccess and delete everything above "# BEGIN WordPress".

    Also check to ensure that wordpress is allowing search engines to go to your site (check Settings -> Privacy in your wordpress admin, by default it blocks them).

    Make sure to test that your site is running after you change the .htaccess file, it should be fine but you never know. If it breaks, then move htaccess.bak to .htaccess to bring it back online.

    Where are you getting these errors from? Google Webmaster Tools? If so, you might need to wait a few days or week for them to update the reports once you put these changes in.


    That Auth stuff is for SSH authentication keys, so you might want to keep those lines.
    {{ DiscussionBoard.errors[3399730].message }}
    • Profile picture of the author vbskss
      Hi Christian Little,

      Thanks for the fast reply i did what you told me the site is still working any idea what this was for and my privacy settings are set to allow search engine to see the website.

      And yes i am getting this error from google webmaster tool but the thing is that i had this error the 13/02 then i added a robots.txt file and waited for the bots to update and they did but still got the errors even after using the tool in webmaster to see if my robots.txt file was allowing them to fetch my website.

      Anyway hope this will work gona try to find out what those lines are for thanks again.
      {{ DiscussionBoard.errors[3399788].message }}
  • Profile picture of the author Christian Little
    Wait a few days to a week, if GWT is still giving you the errors drop me an email and I'll help you further - chris@boltwebhosting.com

    Good luck
    {{ DiscussionBoard.errors[3399835].message }}

Trending Topics