Need help please regarding Webmaster Tools results

by 8 replies
9
Hi warrior, hope one of you guys will able to advice me.


I was checking the "Health" of my sites in Webmaster Tools and noticed that two sites have some blocked URLs. One site has 4 blocked URLs and the other 11. I did not know how to find out why or which URLs are affected and I contacted my web host provider, Hostgator. Their security team did some checking.

This is the response I got from them:

Would you please contact Google Support and ask them to clarify what the 4 blocks referenced in the report were pulled from? I was not able to find a robots.txt file on the accounts:
[root@gatorXXXX /home/myusername/public_html]# find . -type f -name robots.txt
[root@gatorXXXX /home/myusername/public_html]#

[root@gatorYYYY /home/myusername/public_html]# find . -type f -name robots.txt
[root@gatorYYYY /home/myusername/public_html]#

Could anyone advise me what to do? I posted this question on Google webmaster tools forum about 5 days ago but did not get any response at all. So now I deleted the question from there hoping that one of you will be able to help me understand this issue.

Thanks,

Dita
#programming #results #tools #webmaster
  • when you said the url is blocked, it maybe your robots.txt block the url, check your robots.txt content
  • If you talking about that 2 sites from your signature than both of them have this in robots.xtx:

    Disallow: /wp-admin/
    Disallow: /wp-includes/

    This mean that all URLs which start from this name are blocked for indexing so Google can report that. But if only such URLs are blocked than this is fine and nothing to fix. Problem can be if other URLs with valuable content are blocked which should be indexed in Google. But for sure robots.txt do not block anything more than admin and internal WP URLs.

    Tom
  • You are probably using a plugin to dynamically create a robots.txt file, and hostgator is looking for an actual robots.txt file, so their answer is accurate as far as it goes.

    It is ok to block the googlebot from some url's like:
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-login.php
    Disallow: /wp-register.php
  • If in Webmaster Tools an "URL is blocked" is appearing then it is just Google Bot that can not crawl the URL.

    In most cases it is a problem with robots.txt ... when using any SEO plugin you should also check these sections regarding robots.txt or Bots!

    Some plugins also block "bad bots"...
  • Hi warriors,

    Thank you so much for the answers. There is a box under the results and it says (for one of the sites in question)
    http://ebizdame.com/robots.txt content - edit to test changes

    In the box it says:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Sitemap: http://ebizdame.com/sitemap.xml.gz

    The same thing is found for the sites where the URLs are not blocked.

    I was looking at the plugins and they are all similar on all sites with or without blocked URLs. The only common ones between the sites with blocked URLs is the commentluv plugin . Is it possible that some of the blocked URLs may be the comment links???

    Just wondering
  • I do not know the blocked URL's...

    -> http://seohour.com/seo/does-commentl...our-blogs-seo/
  • Well, if we don't know what URLs are blocked than we can't help you more. But you can check what URLs are blocked. Read this article to learn how to do it:

    Understanding Google Webmaster Tools Warnings | Search Engine Academy Blog | Search Engine Academy

    Tom
  • As previously stated, take a look in to your robots.txt, also snoop around in your .htaccess as sometimes you can block certain pages through that file.
Join the discussion

Next Topics on Trending Feed