If I want to block third party bots to crawl my website, what I have to do?

by jenudsouza

Posted: 2 years ago 8 replies

I want to block all third party bots and free seo tool site to crawl my website. It is not possible to block all the bots in Robots.txt and .htaccess. Is is CDN level or Server level to block. Any suggestions

#block #bots #crawl #party #website

Janvi Arora 2 years ago

Hey there,

I have read the following codes that can be used to block bad bots. I hope this will help you.
1. Use the BrowserMatchNoCase directive like this:

BrowserMatchNoCase "agent1" bots
BrowserMatchNoCase "Cheesebot" bots
BrowserMatchNoCase "Catall Spider" bots

Order Allow,Deny
Allow from ALL
Deny from env=bots

2. And here is an example on Nginx:

if ($http_user_agent ~ (agent1|Cheesebot|Catall Spider) ) {
return 403;
}

I haven't tried these codes yet since robots.txt is the best way to block bots. if it works, do let me know.
- Thanks
- Reply
Signature

Want to Future-Proof Your Online Business? Make Your Website SEO Friendly. Click here -> https://www.radon-media.com/services...-optimization/
{{ DiscussionBoard.errors[11482567].message }}
luciesmazanska Banned 2 years ago

you have to do it manually in a google master tools
in settings go to a crawler and change the settings
- Thanks
- Reply
Signature

â˜…â˜…â˜…â˜…â˜…
TARGETED GOOGLE TRAFFIC - CHEAP!
{{ DiscussionBoard.errors[11482667].message }}
kuchenchef 2 years ago

i'm blocking them with the code posted by janvi. in the config file for the virtualhost, htaccess would also work. there are lists of user agents on the internet but i only block those that i see in the server logs.

using the robots.txt doesn't work. some bots ignore the file.
- Thanks
- Reply
{{ DiscussionBoard.errors[11486441].message }}
jamesandersonicb 2 years ago

You can block any third party crawler in a .htaccess file.
- Thanks
- Reply
{{ DiscussionBoard.errors[11487773].message }}
michaelkoehler92 2 years ago

.htaccess should be the best option to do so but you would need the exact codes for the websites that you need to block.

For major websites, it would be easy but smaller sites it could be difficult.
- Thanks
- Reply
Signature

-->How To Get Businesses To Pay You $4,000 - 5,000 Per Month For Ranking Their YouTube Videos!<--
{{ DiscussionBoard.errors[11488066].message }}

MikeFriedman

2 years ago

Originally Posted by jenudsouza

Why in the world would you want to block all 3rd party bots?

Thanks
Reply
1 reply

Signature

Anyone Who is Selling Links Based On DA is Either Incompetent
or They Are a Scammer - Watch the Video

{{ DiscussionBoard.errors[11488074].message }}

michaelkoehler92 2 years ago

Originally Posted by MikeFriedman

Why in the world would you want to block all 3rd party bots?

So that his PBNs won't get tracked by those 3rd party services.
- Thanks
- Reply
- 1 reply
Signature

-->How To Get Businesses To Pay You $4,000 - 5,000 Per Month For Ranking Their YouTube Videos!<--
{{ DiscussionBoard.errors[11488107].message }}
- MikeFriedman 2 years ago
  
  Originally Posted by michaelkoehler92
  
  So that his PBNs won't get tracked by those 3rd party services.
  
  That's just dumb.
  
  First, blocking bots on your website doesn't help. You have to block them on each individual pbn site.
  
  Second, that is a great footprint to give Google that this is probably an SEO network.
  
  [ 1 ] Thanks
  
  Reply
  
  Signature
  
  Anyone Who is Selling Links Based On DA is Either Incompetent
  or They Are a Scammer - Watch the Video
  
  {{ DiscussionBoard.errors[11488156].message }}

If I want to block third party bots to crawl my website, what I have to do?

Trending Topics

Critique My Copy | What am I missing?

Things You Do That Your Family Or Friends Don't Know About

Why do most people quit?

My Wife Said I Can't But Find Help Here....

Ever Battled With Bright Shiny Object Syndrome? How Did You Overcome It?