Blocking bots with htaccss and robots.txt files

10 replies
  • SEO
  • |
Hi

Does anyone know why you should use both the htaccess file and robots.txt to block certain bots from accessing your site instead of just using the robots.txt?

I thought the htaccess was just for internal access.

Thanks
#blocking #bots #files #htaccss #robotstxt
Avatar of Unregistered
  • Profile picture of the author expmrb
    We use only robots.txt file primarily.
    Signature
    SEO Motionz Forum - A Digital Marketing Forum
    Forum Management & Promotion, SEO Tips, Money Making tips etc.
    {{ DiscussionBoard.errors[11491055].message }}
  • Profile picture of the author MatthewWoodward
    You can do it with just .htaccess


    The only difference is .htaccess is server side and can't be read by anyone, including Google or your competitors.

    Where as the robot.txt file is public domain!
    {{ DiscussionBoard.errors[11491145].message }}
    • Profile picture of the author Shaun J
      Originally Posted by MatthewWoodward View Post

      You can do it with just .htaccess


      The only difference is .htaccess is server side and can't be read by anyone, including Google or your competitors.

      Where as the robot.txt file is public domain!

      Ok, understood. So the results are the same whichever method you use, it's just a case of keeping it private.

      Thanks!
      {{ DiscussionBoard.errors[11491369].message }}
      • Profile picture of the author MikeFriedman
        Originally Posted by Shaun J View Post

        Ok, understood. So the results are the same whichever method you use, it's just a case of keeping it private.

        Thanks!
        No. You want to do it with .htaccess file.

        Robots.txt can be ignored. The Robots.txt file is more like a suggestion than a directive. The .htaccess file cannot be ignored.

        Why do you want to block bots though?
        Signature
        SEO, AdWords Management, Social Media Marketing, and more.
        Get a FREE Quote.
        {{ DiscussionBoard.errors[11491476].message }}
        • Profile picture of the author Shaun J
          Originally Posted by MikeFriedman View Post

          No. You want to do it with .htaccess file.

          Robots.txt can be ignored. The Robots.txt file is more like a suggestion than a directive. The .htaccess file cannot be ignored.

          Why do you want to block bots though?
          Mainly to keep competitors from crawling the site.
          {{ DiscussionBoard.errors[11491488].message }}
          • Profile picture of the author expmrb
            Originally Posted by Shaun J View Post

            Mainly to keep competitors from crawling the site.

            Your competitors has bots that can crawl?
            Signature
            SEO Motionz Forum - A Digital Marketing Forum
            Forum Management & Promotion, SEO Tips, Money Making tips etc.
            {{ DiscussionBoard.errors[11491492].message }}
            • Profile picture of the author Shaun J
              Originally Posted by expmrb View Post

              Your competitors has bots that can crawl?
              No, competitors will use crawlers.
              {{ DiscussionBoard.errors[11491506].message }}
          • Profile picture of the author MikeFriedman
            Originally Posted by Shaun J View Post

            Mainly to keep competitors from crawling the site.
            Crawling the site to find what exactly?
            Signature
            SEO, AdWords Management, Social Media Marketing, and more.
            Get a FREE Quote.
            {{ DiscussionBoard.errors[11491515].message }}
  • It is used to prevent the site from indexing the unwanted sites.
    {{ DiscussionBoard.errors[11494333].message }}
Avatar of Unregistered

Trending Topics