12 {{ upvoteCount | shortNum }}
12 {{ upvoteCount | shortNum }}

fake googlebot?

by behgez
10 replies
I have some serious bandwidth problem on my website/hosting account, but I cannot tie all the details together. Maybe someone has experience with this.

1. According to the host's webalytics a few of my sites are getting hundreds of visitors. According to Google Analytics, these sames sites get between 2 and 20 visitors per day (instead of hundreds).

2. According to the host's webalytics, 95% to 98% of the hits are caused by agents/bots, of which the main one has the name Googlebot. According to Google Webmastertools, the spider only crawls about 25K per dag and just a few thousand hits per day (which is only a few hits per minute).

Could it be fake bots that are attaching my site under the Googlebot name? How can I make sure (e.g. via robots.txt) that only real Bots are visiting my sites? Any other ideas or experience with this is welcome!
#fake #googlebot
  • Profile picture of the author Juan Esteban
    Which host are you using? Their analytics may not be accurate.

    You could be suffering a DDOS attack. Well, I am not an expet at this but I think that's the way those attacks work by simulating visitors to your site

    Originally Posted by behgez View Post

    I have some serious bandwidth problem on my website/hosting account, but I cannot tie all the details together. Maybe someone has experience with this.

    1. According to the host's webalytics a few of my sites are getting hundreds of visitors. According to Google Analytics, these sames sites get between 2 and 20 visitors per day (instead of hundreds).

    2. According to the host's webalytics, 95% to 98% of the hits are caused by agents/bots, of which the main one has the name Googlebot. According to Google Webmastertools, the spider only crawls about 25K per dag and just a few thousand hits per day (which is only a few hits per minute).

    Could it be fake bots that are attaching my site under the Googlebot name? How can I make sure (e.g. via robots.txt) that only real Bots are visiting my sites? Any other ideas or experience with this is welcome!
    {{ DiscussionBoard.errors[3089886].message }}
    • Profile picture of the author behgez
      Its a Dutch provider (coloprovider) and I have a VPS with them. It is not a one time thing, but continues workload on the server. I'm sure it is a bot, but I just want to make sure its Google (despite webmastertools says its not).
      {{ DiscussionBoard.errors[3090059].message }}
      • Profile picture of the author Ambius
        if you can read the detailed access log and find that 90% of your traffic is coming from a single IP address then you can figure out where that IP is from and possibly ban it.
        {{ DiscussionBoard.errors[3090217].message }}
        • Profile picture of the author Juan Esteban
          Originally Posted by Ambius View Post

          if you can read the detailed access log and find that 90% of your traffic is coming from a single IP address then you can figure out where that IP is from and possibly ban it.
          This is the first you should do. Very good advice.

          then contact your hosting. You are his customer so they HAVE to help you. If they don't then shift to another one
          {{ DiscussionBoard.errors[3090250].message }}
          • Profile picture of the author AlTheGr8
            It's very easy to write a bot and fake its user agent information, so it will look as a googlebot to your site. Somewhere online I have seen a range of IPs that the real google bot uses. Anything outside of that range can be safely considered fake and banned.
            {{ DiscussionBoard.errors[3092990].message }}
  • Profile picture of the author mihir
    Even if it's genuine Google Bot, you can change crawling rate for your website from Google Webmaster Tools.
    {{ DiscussionBoard.errors[3103378].message }}
  • Profile picture of the author Igal Zeifman
    Hi
    Our company recently released a study of fake Googlebot behavior that shows an average of 21% for fake Googlebot visits (75% of these were directly harmful)

    Read more here: Fake Googlebot study

    To prevent Googlebot impersonation attempts, you can use this Validation technique suggested by Google.

    Also, if these are not fake visits, you can control the visit frequency via WMT.
    {{ DiscussionBoard.errors[6656310].message }}
  • Profile picture of the author GforceSage
    You must kill the fake Googlebots, before they kill your business.

    Also, if you can trace the IP address to a person in the U.S.(doubtful)< they owe you for lost revenue.
    {{ DiscussionBoard.errors[6656377].message }}
  • Profile picture of the author oknoorap
    This is my answer based my experience :

    1. Check bot's IP address (if their IP address is not match with google/other SE, then it's bot).

    2. Maybe someone steal your contents, yeah unbelievable right?
    They harvesting your content, yes I have done that, and creates PHP script to crawling other website is very easy. Be careful. Back to step #1
    {{ DiscussionBoard.errors[6670360].message }}
  • Profile picture of the author jaasmit
    If you have total control over it then you can do that.
    Or you can give some complains to the google team.
    {{ DiscussionBoard.errors[6698840].message }}

Trending Topics