10 {{ upvoteCount | shortNum }}
10 {{ upvoteCount | shortNum }}

Google webmaster question on robots.txt

by mileagedriver
8 replies
  • SEO
    • |
My website was running good without any issue but suddenly on Sep 12th, it says "Googlebot cannot access your site" and the error percentage was 33%. Will that affect my website ?

Also..if googlebot can't crawl how can it show me visitors on google analytics !!

They both are different ?

Please clarify me!
#google #question #robotstxt #webmaster
  • Profile picture of the author paulgl
    The googlebot not being able to crawl your site does
    not mean others can't view it.

    Most of the time, if the bot can't crawl it, then
    nobody can. But visitors can view the cache. and maybe
    at certain points they are let in.

    robots.txt has nothing to do with the googlebot not crawling
    your site.

    Google looks for the robots.txt file first, so if not found,
    that's the first thing they list. But that does not mean if
    they do not find a robots.txt then they don't crawl it.
    I have no robots.txt on the majority of my sites.

    If google gives that error, it means they cannot access the
    entire site, period. It's a server or hosting issue, not
    a robots.txt issue.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[7001840].message }}
  • Profile picture of the author JamesColin
    Banned
    If nothing has changed in your robots.txt then fine. But what if somehow a disallow: googlebot has been added to your site's robots.txt?
    Just look at it and look for any mention of googlebot. You can look at it from google webmaster tools if you want. You could also use the tool "fetch as google" from the same place, and aks to look at yoursite.com/robots.txt, JUST IN CASE.
    I don't know anything about your site, but I can think of things to do with .htaccess, server script and robots.txt to screw someone else rankings, while keeping the robots.txt file intact to any outside eyes but googlebot, if I have access to his server's ftp.
    {{ DiscussionBoard.errors[7003494].message }}
  • Profile picture of the author mileagedriver
    Thanks Paul & James.
    {{ DiscussionBoard.errors[7004340].message }}
  • Profile picture of the author franktwin
    Hi,

    I have found out the hard way, so I will try and make this as simple as I can... You should submit your web site to google web master tools, sign in with your google account and submit your site - they have all sorts of metrics were you can test your site, index it and it comes straight from google, they also have a health section were you can see how well your site is optimized and if it is performing well in google search. I highly recommend it for newbies!
    Signature

    A Premier and Unique Dating Experience | NosyDating.Com

    {{ DiscussionBoard.errors[7004349].message }}
  • Profile picture of the author TheContentAuthority
    Mileagedriver,

    Robots.txt is an optional file. Its purpose is to instruct search engines (and other bots) on which pages you want them to crawl. Note that a program can be made to ignore this file. Google and other major search engines abide by these files. If there is no reason to block them from some pages of your site, I would suggest just deleting the file.

    Also note that if Googlebot cannot crawl your site, regular visitors will still be able to. You could block Google from indexing your site completely and people would never know the difference.

    Your Google Analytics code will still execute whether you have blocked Google or not. This is completely separate from robots.txt. The only reason Analytics would not execute is if a visitor had JavaScript disabled in their browser or if you had it installed on the bottom of a page and the visitor left the page before the script could execute.

    Here is an article on the basics of robots.txt,

    What is Robots.txt

    Hope that helps,

    Shawn
    Signature
    Outsource to the experts...

    We customize your Blog, eBook, Press Release and Sale Copy content with your message.

    {{ DiscussionBoard.errors[7007903].message }}
  • Profile picture of the author Igal Zeifman
    HI

    Agree with @TheContentAuthority, robots.txt is optional, but still a "nice to have".

    However, after experiencing a similar issue myself, I think you should be more concerned with an availability factor, because this may indicate downtime issues.

    Keep in mind, 33% is pretty high and prolonged/frequent downtime issues can and will hurt your SEO (among other things, such as repeat visits, trust and etc)

    I would recommend using Pingdom (or similar) to monitor you websites availability.
    {{ DiscussionBoard.errors[7012250].message }}
  • Profile picture of the author moccc
    i have same problem here
    {{ DiscussionBoard.errors[7012288].message }}
  • Profile picture of the author Ranksword
    use robots.txt configured properly on your website.
    This will mostly solve.
    Signature
    Check your website worth - webvaluecheck.com
    {{ DiscussionBoard.errors[7012495].message }}

Trending Topics