9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

If I want to block third party bots to crawl my website, what I have to do?

by jenudsouza
8 replies
  • SEO
    • |
I want to block all third party bots and free seo tool site to crawl my website. It is not possible to block all the bots in Robots.txt and .htaccess. Is is CDN level or Server level to block. Any suggestions
#block #bots #crawl #party #website
Avatar of Unregistered
  • Profile picture of the author Janvi Arora
    Hey there,

    I have read the following codes that can be used to block bad bots. I hope this will help you.
    1. Use the BrowserMatchNoCase directive like this:

    BrowserMatchNoCase "agent1" bots
    BrowserMatchNoCase "Cheesebot" bots
    BrowserMatchNoCase "Catall Spider" bots

    Order Allow,Deny
    Allow from ALL
    Deny from env=bots

    2. And here is an example on Nginx:

    if ($http_user_agent ~ (agent1|Cheesebot|Catall Spider) ) {
    return 403;
    }

    I haven't tried these codes yet since robots.txt is the best way to block bots. if it works, do let me know.
    Signature

    Want to Future-Proof Your Online Business? Make Your Website SEO Friendly. Click here -> https://www.radon-media.com

    {{ DiscussionBoard.errors[11482567].message }}
  • Profile picture of the author luciesmazanska
    you have to do it manually in a google master tools
    in settings go to a crawler and change the settings
    Signature
    {{ DiscussionBoard.errors[11482667].message }}
  • Profile picture of the author kuchenchef
    i'm blocking them with the code posted by janvi. in the config file for the virtualhost, htaccess would also work. there are lists of user agents on the internet but i only block those that i see in the server logs.

    using the robots.txt doesn't work. some bots ignore the file.
    {{ DiscussionBoard.errors[11486441].message }}
  • Profile picture of the author jamesandersonicb
    You can block any third party crawler in a .htaccess file.
    {{ DiscussionBoard.errors[11487773].message }}
  • Profile picture of the author michaelkoehler92
    .htaccess should be the best option to do so but you would need the exact codes for the websites that you need to block.

    For major websites, it would be easy but smaller sites it could be difficult.
    Signature
    {{ DiscussionBoard.errors[11488066].message }}
  • Profile picture of the author MikeFriedman
    Originally Posted by jenudsouza View Post

    I want to block all third party bots and free seo tool site to crawl my website. It is not possible to block all the bots in Robots.txt and .htaccess. Is is CDN level or Server level to block. Any suggestions
    Why in the world would you want to block all 3rd party bots?
    Signature
    SEO, AdWords Management, Social Media Marketing, and more.
    Get a FREE Quote.
    {{ DiscussionBoard.errors[11488074].message }}
Avatar of Unregistered

Trending Topics