19 {{ upvoteCount | shortNum }}
19 {{ upvoteCount | shortNum }}

Protection from Alexa

by Asher
17 replies
Hi Warriors,

Can someone enlighten me on how to disallow Alexa's bots from crawling
and indexing my pages using robots.txt or some other method?

I understand that Alexa doesn't really honour the "nofollow, noindex"
code.

For those who have products, this is especially important to know as
well.

Asher
#alexa #protection
  • Profile picture of the author dvduval
    Well, since much of Alexa's data is a result of their toolbar that is installed on the user's browser, we are not dealing with a spider in the traditional sense, and it may not be possible to do a whole lot to stop them from collecting data about your site.
    Signature
    It is okay to contact me! I have been developing software since 1999, creating many popular products like phpLD.
    {{ DiscussionBoard.errors[223111].message }}
    • Profile picture of the author Asher
      Originally Posted by dvduval View Post

      Well, since much of Alexa's data is a result of their toolbar that is installed on the user's browser, we are not dealing with a spider in the traditional sense, and it may not be possible to do a whole lot to stop them from collecting data about your site.
      I know it's possible - otherwise, all anyone needs to do is go to Alexa
      with the right search terms and they can get everything that they want!

      What I don't know is how it's done and what are the best ways and
      methods to protect my stuff.

      The best solution is definitely to buy a good defense script like DLGuard
      (if it's a script).

      Thing is, how do I just prevent Alexa from getting their grubby fingers
      on my downloads =D

      Asher
      Signature
      Asher Aw
      {{ DiscussionBoard.errors[223136].message }}
  • Profile picture of the author TheRichJerksNet
    Hi Asher,
    Could you explain in more details exactly what you issues is please..

    James
    {{ DiscussionBoard.errors[223140].message }}
  • Profile picture of the author Asher
    Well, from my understanding, Alexa doesn't follow
    the meta tag for nofollow, noindex.

    So any webpage that I have which has a download
    link on it will get indexed and recorded into Alexa

    I don't want that to happen - so what can I do to
    prevent my pages from getting indexed and recorded?

    Asher
    Signature
    Asher Aw
    {{ DiscussionBoard.errors[223170].message }}
    • Profile picture of the author TheRichJerksNet
      Originally Posted by Asher View Post

      Well, from my understanding, Alexa doesn't follow
      the meta tag for nofollow, noindex.

      So any webpage that I have which has a download
      link on it will get indexed and recorded into Alexa

      I don't want that to happen - so what can I do to
      prevent my pages from getting indexed and recorded?

      Asher
      Asher,
      Alexa is by far not the only bot you need to worry about..There are many bad bots online designed to seek out your download links.

      Maybe you should go to my profile, view threads started by me and find my WordPress Secured WSO .. Inside that package I include a BadBot-Killer script that will allow you to block as manybad bots as you want. I even have it pre set with like 20 or 30 but you can add more if you want..

      Very effective script that I coded.. simple but powerful..

      James
      {{ DiscussionBoard.errors[223300].message }}
  • Profile picture of the author Paul1234
    User-agent: ia_archiver
    Disallow: /

    Make that the first entry in a robots.txt file in your root directory. This will prevent the entire site from being indexed by Alexa.
    Signature

    Paul Turner

    {{ DiscussionBoard.errors[223291].message }}
  • Profile picture of the author Mary Green
    Why not use e-junkie for now until you can get DL Guard. With e-junkie download links are only good for as many hours as you choose. But I have to say DL Guard is the best. E-junkie is $5 a month.
    Signature
    Free Guide- 52 Time Saving Social Media Tools on Social Media Fuze
    {{ DiscussionBoard.errors[223306].message }}
  • Profile picture of the author Hendry Lee
    Use a protection script... Alexa is known not to obey robots.txt, although the line from Paul helps.
    Signature
    Syndicative e-Letter: NOT Just Another Newsletter With Random Content
    {{ DiscussionBoard.errors[223393].message }}
    • Profile picture of the author TheRichJerksNet
      Originally Posted by Hendry Lee View Post

      Use a protection script... Alexa is known not to obey robots.txt, although the line from Paul helps.
      I would agree.. My BadBot-Killer Script haspre-set to block the following.. And there are easy admin controls to add/delete more bots..

      LibWWW
      Zeus
      Xaldon WebSpider
      Wget
      WebStripper
      Website eXtractor
      WebSauger
      WebReaper
      WebFetch
      WebAuto
      WebCopier
      Teleport Pro
      SuperHTTP
      SuperBot
      SmartDownload
      SiteSnagger
      RealDownload/4.0.0.40
      RealDownload
      NetZi
      NetAnts
      Net Vampire/3.0
      Mister Pix II 2.01
      Mass Downloader/2.2
      larbin
      larbin_2.6.
      Internet Ninja 4.0
      HTTrack
      Go!Zilla
      GetRight
      Exabot/3.0
      Gigabot/2.0
      WebZIP
      Snoopy
      libwww-perl

      Most do not realize but there are many bots out there searchingyour server for information and download links..

      See my signature as I have this script included in WordPress Secured Digital Report and the script can be used on any website or server you have.. It is not a wp plug-in .. It is a stand alone php script..

      James
      {{ DiscussionBoard.errors[223408].message }}
  • Profile picture of the author Chris Lockwood
    Don't the bad bots just ignore robots.txt and meta tags?
    Signature
    Comprehensive membership site training
    My other blog
    {{ DiscussionBoard.errors[223560].message }}
    • Profile picture of the author TheRichJerksNet
      Originally Posted by Chris Lockwood View Post

      Don't the bad bots just ignore robots.txt and meta tags?
      Hi Chris,
      Most of the time, yes they do...

      This is why I created a php script to block them, my script does not use robots.txt ..

      James
      {{ DiscussionBoard.errors[223573].message }}
  • Profile picture of the author Jesus Perez
    Alexa finds your product via the toolbar. It cannot be blocked.

    What you need to do is protect your products and download links. DLGuard can do this (What I use). e-Junkie can also. If you hit RetailMeNot, you'll find some coupons for e-Junkie.
    Signature

    {{ DiscussionBoard.errors[223577].message }}
    • Profile picture of the author TheRichJerksNet
      Originally Posted by BlueSquares View Post

      Alexa finds your product via the toolbar. It cannot be blocked.

      What you need to do is protect your products and download links. DLGuard can do this (What I use). e-Junkie can also. If you hit RetailMeNot, you'll find some coupons for e-Junkie.
      Actually Alexa can be blocked.. I have done so one one of my sites. Now yep you end up blocking some others too but oh well, not worried about blocking them..

      James
      {{ DiscussionBoard.errors[223612].message }}
      • Profile picture of the author Jesus Perez
        Originally Posted by TheRichJerksNet View Post

        Actually Alexa can be blocked.. I have done so one one of my sites. Now yep you end up blocking some others too but oh well, not worried about blocking them..

        James
        James,

        Really? Are you blocking based on the IP range? And won't blocking the toolbar also block the buyer from purchasing?

        I smell a WSO.
        Signature

        {{ DiscussionBoard.errors[223767].message }}
        • Profile picture of the author TheRichJerksNet
          Originally Posted by BlueSquares View Post

          James,

          Really? Are you blocking based on the IP range? And won't blocking the toolbar also block the buyer from purchasing?

          I smell a WSO.
          I do have another script (which I coded) which I can use IP Blocking by Range and Country.. The one website that I use to block alexa also have 72 countries and over 6,000 ip's blocked.. There is a reason behind this and yes it may block some buyers but not enough for me to worry about..

          So what kind of WSO you smell ..lol

          James
          {{ DiscussionBoard.errors[223858].message }}
  • Profile picture of the author Asher
    Thanks for the excellent information shared. In that case,
    would it be a good idea to only have the download links by email?

    If so, is there also codes to prevent the indexing and recording of
    .zip, .exe, .pdf, etc files?

    Again, I saw a code somewhere about that so I'm asking.

    Asher
    Signature
    Asher Aw
    {{ DiscussionBoard.errors[225268].message }}

Trending Topics