8 replies
  • SEO
  • |
What is robots.txt?
How it connects with SEO?
#robotstxt
  • Profile picture of the author dave147
    [DELETED]
    {{ DiscussionBoard.errors[1115677].message }}
    • Profile picture of the author Jimerson Farveez
      Originally Posted by dave147 View Post

      my understanding of it is: it's a piece of code inserted onto your site, mainly the download page that tells the SE spiders not to follow the page and index it on the SE listings
      Thxs, Do you have any ideas on how to add this code? Can I have a sample code please?
      Signature
      Search Engine Optimization Services in Delray Beach, FL
      {{ DiscussionBoard.errors[1115685].message }}
      • Profile picture of the author Agnel
        Originally Posted by Jimerson Farveez View Post

        Thxs, Do you have any ideas on how to add this code? Can I have a sample code please?
        User-Agent: *
        Allow: /
        {{ DiscussionBoard.errors[1116264].message }}
        • Profile picture of the author TristanPerry
          Originally Posted by Agnel View Post

          User-Agent: *
          Allow: /
          Yep, that's one of the best ones. It basically allows everything. I tend to use the following which is a bit different:

          User-agent: *
          Disallow: /cgi-bin/
          Allow: /

          Sitemap: http://www.sitenamehere.com/sitemap.xml
          That blocks the cgi-bin, just in-case. Also having the sitemap parameter there is useful as Google et al sometimes then spider your sitemap XML file.

          (I then generate the XML sitemaps via Cogah - XML Sitemap Generator)
          Signature
          Plagiarism Guard - Protect Against Content Theft
          {{ DiscussionBoard.errors[1116278].message }}
  • Profile picture of the author Liquification
    Actually it's a file that is uploaded to your hosting site that tells search engine spiders what pages to crawl and not crawl. I don't have a sample off hand but, unless you have pages on your site that you don't want crawled you really don't need one.
    Signature

    Get your FREE Site Building Report HERE

    Money Making Tips at Making Money With Google

    {{ DiscussionBoard.errors[1116043].message }}
  • Profile picture of the author yuyuan
    A sample code of robots.txt:

    User-agent: *
    Disallow: /folder1/
    Disallow: /folder2/
    Disallow: /folder3/
    Disallow: /folder4/
    Disallow: /folder5/

    Place this code in a notepad and save the notepad as robots.txt. Then upload the .txt file to your root directory.
    {{ DiscussionBoard.errors[1116187].message }}
  • Profile picture of the author TristanPerry
    A robots.txt file is a text file (with a particular syntax) which tells the search engine bots (and other bots) which parts of your site they can/cannot access.

    I've created a free (and unlimited, for now) robots.txt generator at:

    Cogah - Robots.txt Generator

    Once it's generated, download it and then upload it to your website's base/root directory (so it can be accessed at http://www.example.org/robots.txt)
    Signature
    Plagiarism Guard - Protect Against Content Theft
    {{ DiscussionBoard.errors[1116258].message }}

Trending Topics