11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

Blocking bots with htaccss and robots.txt files

by Shaun J
10 replies
  • SEO
    • |
Hi

Does anyone know why you should use both the htaccess file and robots.txt to block certain bots from accessing your site instead of just using the robots.txt?

I thought the htaccess was just for internal access.

Thanks
#blocking #bots #files #htaccss #robotstxt
Avatar of Unregistered
  • Profile picture of the author expmrb
    We use only robots.txt file primarily.
    Signature
    SEO Motionz Forum & Blog- Digital Marketing Forum & Blog,
    Forum Management & Promotion, SEO Tips, Money Making tips etc.
    {{ DiscussionBoard.errors[11491055].message }}
  • Profile picture of the author MatthewWoodward
    You can do it with just .htaccess


    The only difference is .htaccess is server side and can't be read by anyone, including Google or your competitors.

    Where as the robot.txt file is public domain!
    Signature
    {{ DiscussionBoard.errors[11491145].message }}
    • Profile picture of the author Shaun J
      Originally Posted by MatthewWoodward View Post

      You can do it with just .htaccess


      The only difference is .htaccess is server side and can't be read by anyone, including Google or your competitors.

      Where as the robot.txt file is public domain!

      Ok, understood. So the results are the same whichever method you use, it's just a case of keeping it private.

      Thanks!
      {{ DiscussionBoard.errors[11491369].message }}
      • Profile picture of the author MikeFriedman
        Originally Posted by Shaun J View Post

        Ok, understood. So the results are the same whichever method you use, it's just a case of keeping it private.

        Thanks!
        No. You want to do it with .htaccess file.

        Robots.txt can be ignored. The Robots.txt file is more like a suggestion than a directive. The .htaccess file cannot be ignored.

        Why do you want to block bots though?
        {{ DiscussionBoard.errors[11491476].message }}
        • Profile picture of the author Shaun J
          Originally Posted by MikeFriedman View Post

          No. You want to do it with .htaccess file.

          Robots.txt can be ignored. The Robots.txt file is more like a suggestion than a directive. The .htaccess file cannot be ignored.

          Why do you want to block bots though?
          Mainly to keep competitors from crawling the site.
          {{ DiscussionBoard.errors[11491488].message }}
          • Profile picture of the author expmrb
            Originally Posted by Shaun J View Post

            Mainly to keep competitors from crawling the site.

            Your competitors has bots that can crawl?
            Signature
            SEO Motionz Forum & Blog- Digital Marketing Forum & Blog,
            Forum Management & Promotion, SEO Tips, Money Making tips etc.
            {{ DiscussionBoard.errors[11491492].message }}
            • Profile picture of the author Shaun J
              Originally Posted by expmrb View Post

              Your competitors has bots that can crawl?
              No, competitors will use crawlers.
              {{ DiscussionBoard.errors[11491506].message }}
          • Profile picture of the author MikeFriedman
            Originally Posted by Shaun J View Post

            Mainly to keep competitors from crawling the site.
            Crawling the site to find what exactly?
            {{ DiscussionBoard.errors[11491515].message }}
  • Profile picture of the author 1digitalmarketing
    It is used to prevent the site from indexing the unwanted sites.
    {{ DiscussionBoard.errors[11494333].message }}
Avatar of Unregistered

Trending Topics