are bad spiders really a problem?

by karma8
14 replies
I've gotten a couple of emails re a product being sold to remove or "spank" the bad spiders that are taking up lots of bandwidth and not adding value to your business, freeing up space for real visitors and not causing a problem with hosting limits. It's called Spyder spanker at that name .com if you want to see the sales page.

Anyway, I'm not sure if this is something useful that I need or not. I do see a lot of spider activity in my stats, but I always thought that was sort of good b/c it means they are crawling my sites and hopefully indexing them.

How do you know how much is too much. I've never had a problem with bandwidth at my hosting accts with spiders, just video. I've had inode issues, but that's different I believe.

Can anyone shed some light on this and let me know your opinion as to whether this is just a scare tactic to sell a product we don't really need but lines the sellers pockets with moolah or if it really could be a useful plugin to have and prevent bandwidth issues.

Thanks for any help/insight!

Sandra
P.S. There was nothing in the review thread.
#bad #bandwidth issues #problem #spiders #spyder spanker
  • Profile picture of the author JohnMcCabe
    Sandra, spiders like googlebot and the other search engine spiders are mostly a good thing. On the other hand, you have spiders looking for anything from content/emails/urls to scrape to weaknesses in your setup that can be exploited. These should be blocked.

    I doubt you need any kind of special program to do it, a robots.txt file should do the ticket. If the spiders ignore that, your host should be able to tweak your server settings to block spiders you don't want. Blocking them is in the host's best interest as well as your own, so I've always found them cooperative.
    {{ DiscussionBoard.errors[6328066].message }}
  • Profile picture of the author sbucciarel
    Banned
    Interesting product. I'll wait until I hear something from some spyder experts.
    {{ DiscussionBoard.errors[6328191].message }}
  • Profile picture of the author karma8
    Thanks for your reply John

    I guess their premise is that the bad spiders ignore the robots.txt file. I do use Hostgator for most hosting and they are very helpful. How do I know if any of the spiders are bad and stealing any info so I can tell hostgator to block them?

    Horny Devil
    Aauugh! You are scaring me! I'm arachnophobic!
    Signature

    Treat others the way you want to be treated. :)
    Check out my updated recipe site and the recent post for easy chewy chocolate brownies...mmm...
    If you want to get a Linkvana subscription, I offer a $30 rebate every month here - Linkvana Discount. Make sure you email your receipt for the rebate b/c I have other promotions that don't offer the rebate. Thx

    {{ DiscussionBoard.errors[6328493].message }}
  • Profile picture of the author Dimachaerus
    I want to know this too, I am going to follow this post.
    {{ DiscussionBoard.errors[6329007].message }}
    • Profile picture of the author Richard Van
      Originally Posted by Horny Devil View Post

      In Oz they are.

      I have family in Sydney and I've always said, if I move out there I will deal with things like that in the following way. I will have a cupboard and in that cupboard will be a sawn of shotgun, bag of plaster and matching paint for the wall.

      I shall simply shoot them and plaster and repaint the wall. I wouldn't go near one of those with anything less!

      Originally Posted by sbucciarel View Post

      Interesting product. I'll wait until I hear something from some spyder experts.
      I bought the product you're talking about, I'm testing it today, it certainly looks interesting and good for technophobes .
      Signature

      Wibble, bark, my old man's a mushroom etc...

      {{ DiscussionBoard.errors[6329725].message }}
  • Profile picture of the author KirkMcD
    Check your logs for bad bot activity.
    Unless you have a big site, it's not something you really need to worry about.
    {{ DiscussionBoard.errors[6330715].message }}
    • Profile picture of the author JohnMcCabe
      Originally Posted by karma8 View Post

      Thanks for your reply John

      I guess their premise is that the bad spiders ignore the robots.txt file. I do use Hostgator for most hosting and they are very helpful. How do I know if any of the spiders are bad and stealing any info so I can tell hostgator to block them?
      As Kirk said, check your logs.

      The big danger is stealing your bandwidth. Some of the spiders sent by spammers will hammer your site as fast as they can, slowing down response for your human visitors.

      I don't have a current list of Search Engine bots/spiders, but a quick check on Google should find you one.
      {{ DiscussionBoard.errors[6333924].message }}
  • Profile picture of the author Ettienne
    They are, yes. To take care of these spam bots, install a FREE wordpress plugin called "Bad Behavior". It's better than most paid products I've seen.

    PS. That's one helluva'n ugly spider!
    {{ DiscussionBoard.errors[6348283].message }}
    • Profile picture of the author karma8
      Originally Posted by Ettienne View Post

      They are, yes. To take care of these spam bots, install a FREE wordpress plugin called "Bad Behavior". It's better than most paid products I've seen.

      PS. That's one helluva'n ugly spider!

      Thanks! Very helpful info. I'll try it on any of my sites that look like they are being zapped or should I say bitten by spyders.........
      Signature

      Treat others the way you want to be treated. :)
      Check out my updated recipe site and the recent post for easy chewy chocolate brownies...mmm...
      If you want to get a Linkvana subscription, I offer a $30 rebate every month here - Linkvana Discount. Make sure you email your receipt for the rebate b/c I have other promotions that don't offer the rebate. Thx

      {{ DiscussionBoard.errors[6349342].message }}
  • Profile picture of the author travlinguy
    Spyder spanking... Sounds kinda kinky.
    {{ DiscussionBoard.errors[6348301].message }}
  • Spiders are bad, mkay?


    Signature
    Philly Web Services - SEO, Internet Traffic, Reputation Management, Website Design, & PPC Management.
    {{ DiscussionBoard.errors[6348422].message }}
    • Profile picture of the author Powder_Skier
      I had a web site recently that was using tons of bandwidth (in the health niche), but never any opt-ins. Aside from the fact that I may have had a crappy opt-in page, I always wondered about the huge usage. So I bought just to be on the safe side, even though it was pricey.
      Signature
      Don't Be A Hackers Victim - Secure Your Wordpress Website Now!
      Not sure the best path to take? Which security plugins are best that won't break your site? Confused about configuring the plugins properly? Don't want to DIY? I'll secure it for you @ a reasonable rate, including backups. Click Here!
      {{ DiscussionBoard.errors[6349677].message }}
  • Profile picture of the author automaton
    There is no such thing as good spiders/bad spiders.
    You can easily manipulate your robots.txt file for free to control the behavior of these bots.
    Check out this 'Robots exclusion standard' entry from wikipedia:
    Robots exclusion standard - Wikipedia, the free encyclopedia
    {{ DiscussionBoard.errors[6349403].message }}

Trending Topics