Permissions for robots.text files

by timpears 8 replies
So I don't want SE spiders looking in one of my folders, so I just uploaded a robots.text file. But I have no idea what permissions I need to put on that file. Doesn't make any sense to have it set to the wrong permission so the spider doesn't get to read it, and I am not sure how to do the permissions for it. Can anyone help me, please?
#main internet marketing discussion forum #files #permissions #robotstext
Avatar of Unregistered
  • Profile picture of the author bradlean
    Hi Im a newbie about robot txt if you don't mind.

    does robot txt allows good search engine ranking?
    {{ DiscussionBoard.errors[3043315].message }}
    • Profile picture of the author KevinTorrence
      I bookmarked this site a while back with all the commands so I could keep them straight.

      Hope it helps:

      Useful HTML Meta Tags - cache, no-cache, robots, refresh, content, keywords, description, expires, author, etc.
      Signature
      {{ DiscussionBoard.errors[3043379].message }}
    • Profile picture of the author timpears
      Originally Posted by bradlean View Post

      does robot txt allows good search engine ranking?
      robots.text is used to stop spiders from crawling certain files and or folders. If you don't want them to be indexed because they are private or membership or what ever, then robots.text can be used to keep the spiders from looking into the folders etc.
      Signature

      Tim Pears

      {{ DiscussionBoard.errors[3043479].message }}
      • Profile picture of the author bradlean
        Originally Posted by timpears View Post

        robots.text is used to stop spiders from crawling certain files and or folders. If you don't want them to be indexed because they are private or membership or what ever, then robots.text can be used to keep the spiders from looking into the folders etc.

        Thank you so much for the reply!

        I really appreciate it.

        Advanced Merry Christmas and A Happy New Year!
        {{ DiscussionBoard.errors[3044041].message }}
      • Profile picture of the author Marvin Johnston
        Originally Posted by timpears View Post

        robots.text is used to stop spiders from crawling certain files and or folders. If you don't want them to be indexed because they are private or membership or what ever, then robots.text can be used to keep the spiders from looking into the folders etc.
        Robots.txt *assumes* that the spider and anyone else who sees it is following the rules. That is not always correct.

        Someone looking at a sites robot.txt file that had some private or membership files listed there wouldn't necessarily stop a human from following those links. And ditto for some web crawlers, especially since the robots.txt file gave them a heads up on what to check out.

        Oops, I just saw that pennynickels made the same point.

        Marvin
        {{ DiscussionBoard.errors[3044297].message }}
  • Profile picture of the author Web Smithy
    Put:
    Code:
    User-agent: *
    Disallow: /yourdirectory/
    in the file... and that's it. And permissions like for any other documents, 644 is just fine.
    {{ DiscussionBoard.errors[3043434].message }}
  • Profile picture of the author medi50cus
    Code:
    User-agent: *
    Disallow: /yourdirectory/
    Use this codes, mate. Hope this helps.
    {{ DiscussionBoard.errors[3043695].message }}
  • Profile picture of the author pennynickels
    Side note:

    Major search engines and respectable small search engines honor the robots.txt file. Non-respectable search engines and some other non-search engines do not and will index the disallowed anyway or just use it to see what you have that you don't want others to know about via those bots.

    There are other schools of thought that says, if there are files or directories you don't want 'known', to not add them to that robots file, but to add the restrictions via e.g. .htaccess instead which is not directly readable by the public.
    {{ DiscussionBoard.errors[3044201].message }}
Avatar of Unregistered

Trending Topics