9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

What is a robot.txt file?

by mdotwhite
8 replies
Hey warriors please help me here.

What is a robots.txt file?

What's it's purpose?

How do I use it?

Is it important for search engine ranking?


Thanks
#file #robottxt
  • Profile picture of the author butters
    This should help.

    The Web Robots Pages
    {{ DiscussionBoard.errors[1600426].message }}
  • Profile picture of the author interplay
    In short the robots.txt file controls what content on your site is indexed by the various search engine 'robots'. Here are a few things that you can allow or disallow with the robots.txt file.

    - What search engine robots can index your site
    - What directories or folders can be indexed
    - What browsers are allowed (aka the user-agent)

    Learn more here: The Web Robots Pages
    Signature
    Internet Marketing Product Launches
    {{ DiscussionBoard.errors[1600457].message }}
    • Profile picture of the author mdotwhite
      Is the robots file necessary for SEO?
      {{ DiscussionBoard.errors[1600585].message }}
      • Profile picture of the author Elle Holder
        Originally Posted by mdotwhite View Post

        Is the robots file necessary for SEO?
        As long as you don't have your robots.txt blocking any page that you want indexed and ranked, no. If you have a page disallowed, all the SEO in the world is not going to help you, since the bots won't crawl it.
        Signature

        {{ DiscussionBoard.errors[1600593].message }}
        • Profile picture of the author seasoned
          Originally Posted by ElleJ View Post

          As long as you don't have your robots.txt blocking any page that you want indexed and ranked, no. If you have a page disallowed, all the SEO in the world is not going to help you, since the bots won't crawl it.

          CORRECTION.... the bots MIGHT NOT crawl it.

          ALSO realize that some pirates will look for the robots.txt file.

          Steve
          {{ DiscussionBoard.errors[1600684].message }}
          • Profile picture of the author Dennis Gaskill
            Originally Posted by seasoned View Post

            CORRECTION.... the bots MIGHT NOT crawl it.

            ALSO realize that some pirates will look for the robots.txt file.

            Steve
            That's true, a few don't honor it, but Google does. A robots.txt file can help with SEO in this way...if you have two pages with the same content, say one is for your visitors to read and the other is a printer friendly page, you can use the robots.txt file to tell the search engines not to index the printer friendly page. In the case of Google, this will keep the page rank from being divided between the two pages.

            Also, be careful about which files you block in the robots.txt file. You may be telling search engine scavengers where you're hiding your goodies.
            Signature

            Just when you think you've got it all figured out, someone changes the rules.

            {{ DiscussionBoard.errors[1600699].message }}
  • Profile picture of the author Brad Lindsay
    A word of warning.

    Make sure you don't list secret pages or secret folders in your robots.txt file. This would
    be a huge security risk as robots.txt files can be read by anyone across the internet. (You should
    consider using a secure download script if you need to protect your valuable downloadable products
    such as smartdd.com or dlguard.com).

    Also for security reasons make sure there is an index.html file in every folder on your site.
    E.g. /images folder which might contain your images should also contain an index.html file
    (this could be just a blank file called index.html or index.htm). Not doing so would allow the
    contents of the folder to be viewed and could also present a security risk.
    {{ DiscussionBoard.errors[1600805].message }}
  • Profile picture of the author danalingga
    Robot.txt is used to command search engine what to do with your site. You can allow or disallow some part of your site regarding to search engine crawler.
    Signature
    Blogging Update
    {{ DiscussionBoard.errors[1600827].message }}

Trending Topics