4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

If I want to block third party bots to crawl my website, what I have to do?

by jenudsouza
3 replies
  • SEO
    • |
I want to block all third party bots and free seo tool site to crawl my website. It is not possible to block all the bots in Robots.txt and .htaccess. Is is CDN level or Server level to block. Any suggestions
#block #bots #crawl #party #website
Avatar of Unregistered
  • Profile picture of the author Janvi Arora
    Hey there,

    I have read the following codes that can be used to block bad bots. I hope this will help you.
    1. Use the BrowserMatchNoCase directive like this:

    BrowserMatchNoCase "agent1" bots
    BrowserMatchNoCase "Cheesebot" bots
    BrowserMatchNoCase "Catall Spider" bots

    Order Allow,Deny
    Allow from ALL
    Deny from env=bots

    2. And here is an example on Nginx:

    if ($http_user_agent ~ (agent1|Cheesebot|Catall Spider) ) {
    return 403;
    }

    I haven't tried these codes yet since robots.txt is the best way to block bots. if it works, do let me know.
    Signature

    Want to Future-Proof Your Online Business? Make Your Website SEO Friendly. Clicke here -> https://www.radon-media.com/services...-optimization/

    {{ DiscussionBoard.errors[11482567].message }}
  • Profile picture of the author luciesmazanska
    you have to do it manually in a google master tools
    in settings go to a crawler and change the settings
    Signature
    {{ DiscussionBoard.errors[11482667].message }}
  • Profile picture of the author kuchenchef
    i'm blocking them with the code posted by janvi. in the config file for the virtualhost, htaccess would also work. there are lists of user agents on the internet but i only block those that i see in the server logs.

    using the robots.txt doesn't work. some bots ignore the file.
    {{ DiscussionBoard.errors[11486441].message }}
Avatar of Unregistered

Trending Topics