Permissions for robots.text files

by timpears 8 replies
So I don't want SE spiders looking in one of my folders, so I just uploaded a robots.text file. But I have no idea what permissions I need to put on that file. Doesn't make any sense to have it set to the wrong permission so the spider doesn't get to read it, and I am not sure how to do the permissions for it. Can anyone help me, please?
#main internet marketing discussion forum #files #permissions #robotstext
Avatar of Unregistered
  • Profile picture of the author bradlean
    Hi Im a newbie about robot txt if you don't mind.

    does robot txt allows good search engine ranking?
    {{ DiscussionBoard.errors[3043315].message }}
    • Profile picture of the author KevinTorrence
      I bookmarked this site a while back with all the commands so I could keep them straight.

      Hope it helps:

      Useful HTML Meta Tags - cache, no-cache, robots, refresh, content, keywords, description, expires, author, etc.
      Signature
      {{ DiscussionBoard.errors[3043379].message }}
    • Profile picture of the author timpears
      Originally Posted by bradlean View Post

      does robot txt allows good search engine ranking?
      robots.text is used to stop spiders from crawling certain files and or folders. If you don't want them to be indexed because they are private or membership or what ever, then robots.text can be used to keep the spiders from looking into the folders etc.
      Signature

      Tim Pears

      {{ DiscussionBoard.errors[3043479].message }}
      • Profile picture of the author bradlean
        Originally Posted by timpears View Post

        robots.text is used to stop spiders from crawling certain files and or folders. If you don't want them to be indexed because they are private or membership or what ever, then robots.text can be used to keep the spiders from looking into the folders etc.

        Thank you so much for the reply!

        I really appreciate it.

        Advanced Merry Christmas and A Happy New Year!
        {{ DiscussionBoard.errors[3044041].message }}
      • Profile picture of the author Marvin Johnston
        Originally Posted by timpears View Post

        robots.text is used to stop spiders from crawling certain files and or folders. If you don't want them to be indexed because they are private or membership or what ever, then robots.text can be used to keep the spiders from looking into the folders etc.
        Robots.txt *assumes* that the spider and anyone else who sees it is following the rules. That is not always correct.

        Someone looking at a sites robot.txt file that had some private or membership files listed there wouldn't necessarily stop a human from following those links. And ditto for some web crawlers, especially since the robots.txt file gave them a heads up on what to check out.

        Oops, I just saw that pennynickels made the same point.

        Marvin
        {{ DiscussionBoard.errors[3044297].message }}
  • Profile picture of the author Web Smithy
    Put:
    Code:
    User-agent: *
    Disallow: /yourdirectory/
    in the file... and that's it. And permissions like for any other documents, 644 is just fine.
    {{ DiscussionBoard.errors[3043434].message }}
  • Profile picture of the author medi50cus
    Code:
    User-agent: *
    Disallow: /yourdirectory/
    Use this codes, mate. Hope this helps.
    {{ DiscussionBoard.errors[3043695].message }}
  • Profile picture of the author pennynickels
    Side note:

    Major search engines and respectable small search engines honor the robots.txt file. Non-respectable search engines and some other non-search engines do not and will index the disallowed anyway or just use it to see what you have that you don't want others to know about via those bots.

    There are other schools of thought that says, if there are files or directories you don't want 'known', to not add them to that robots file, but to add the restrictions via e.g. .htaccess instead which is not directly readable by the public.
    {{ DiscussionBoard.errors[3044201].message }}
Avatar of Unregistered

Trending Topics

  • 2 {{ upvoteCount | shortNum }}

    Giveaways / Freebies: what are yours?

    Connann in Internet Marketing

    We all use giveaways and freebies for building our list in email marketing Today's it seems the uptrend is videos, with the 6 video formula, etc Even if the vast ... [read more]

  • 1 {{ upvoteCount | shortNum }}

    Toys R US filed for Bankruptcy. It lost to E-Commerce. Good News or Bad News for you.

    Amit Patel in eCommerce

    How many of you in the Warrior Forum House are on E-Commerce? Well you have a good news. E-Commerce is the fastest growing business in the world today. And this ... [read more]

  • 1 {{ upvoteCount | shortNum }}

    Are you the Warrior of the Month?

    rankinghero in Internet Marketing

    so.....we are launching the Warrior of the Month mini competition... Yes... every Month we will announce who was the Warrior with the most valuable post. This is our own small ... [read more]

  • 121 {{ upvoteCount | shortNum }}

    Warriors .... Warriors.... what would you like to LEARN?

    Good Morning awesome Warriors... I'm happy to announce that The Warrior Forum is planning to create our first COURSE. Yes... Powered by Warrior Wisdom... yeah!! Made at home. But before ... [read more]

  • 2 {{ upvoteCount | shortNum }}

    is this against CL rules?

    dj2590 in Internet Marketing

    is it against craigslist rules to post my website/ landing page in the offers categories? My front end offer is an affiliate marketing course and the rest of the follow ... [read more]