12 {{ upvoteCount | shortNum }}
12 {{ upvoteCount | shortNum }}

Robots.txt ?

by kevinfar
10 replies
Hi all,

Just a quick question..

robots.txt in the root folder of a site should disallow search engines from indexing folders that you do not want to share with search engines.

However, is there anything that I can do to prevent someone from just typing in http://www.mysite.com/robots.txt ?

This will just bring all of the robots concept into nothing, because this someone will be able to access all of the files.

Thanks for your help.
#robotstxt
  • Profile picture of the author CDarklock
    Originally Posted by kevinfar View Post

    However, is there anything that I can do to prevent someone from just typing in http://www.mysite.com/robots.txt ?
    Nope. The robots.txt file is a polite request. Impolite people won't care.
    Signature
    "The Golden Town is the Golden Town no longer. They have sold their pillars for brass and their temples for money, they have made coins out of their golden doors. It is become a dark town full of trouble, there is no ease in its streets, beauty has left it and the old songs are gone." - Lord Dunsany, The Messengers
    {{ DiscussionBoard.errors[1402695].message }}
    • Profile picture of the author kevinfar
      So, as far as impolite people go, is there any way to make them stay away from valuable files?

      Cheers
      Signature
      >> 12 Free Internet Marketing Tools You Should Be Using <<
      {{ DiscussionBoard.errors[1402707].message }}
      • Profile picture of the author ktmurf
        In your hosting account options, you should have the ability to add password protection to any/all areas of your directory. You can either make a generic username/pass that you give out automatically in receipt redirects or add in the database through (paypal for instance) where they assign new usernames/passwords upon receipt.

        It's all a bit of a pain to setup but once it's going, it's smooth like butta (butter)!

        I use the generic stuff myself. If someone is going to share the info with a friend, they're going to do it no matter what the username/password is.
        {{ DiscussionBoard.errors[1402713].message }}
        • Profile picture of the author kevinfar
          Thanks for the reply Kevin.

          Let's say that you assign password protection to a file called product.html, and to do this you will create the user form in a page called password.html.

          Wouldn't you still need to add product.html to the robots? And will not this defeat all of the purpose of adding password protection?

          Thanks
          Signature
          >> 12 Free Internet Marketing Tools You Should Be Using <<
          {{ DiscussionBoard.errors[1402732].message }}
          • Profile picture of the author DogScout
            Originally Posted by kevinfar View Post

            Thanks for the reply Kevin.

            Let's say that you assign password protection to a file called product.html, and to do this you will create the user form in a page called password.html.

            Wouldn't you still need to add product.html to the robots? And will not this defeat all of the purpose of adding password protection?

            Thanks
            Now, I am no expert and this is right off the cuff, but what if the redirect to a download went to a different site and the password was given to a buyer at a third site. Might that sufficiently confuse a freebie seeker?

            Just a stab in the dark here maybe KT knows better?
            Signature
            {{ DiscussionBoard.errors[1402743].message }}
            • Profile picture of the author kevinfar
              ok just had a look at my hosting, and it seems as if it is a straight forward process to create a password on a particular directory.

              Originally Posted by DogScout

              Re: Robots.txt ?
              Quote:
              Originally Posted by kevinfar
              Thanks for the reply Kevin.

              Let's say that you assign password protection to a file called product.html, and to do this you will create the user form in a page called password.html.

              Wouldn't you still need to add product.html to the robots? And will not this defeat all of the purpose of adding password protection?

              Thanks

              Now, I am no expert and this is right off the cuff, but what if the redirect to a download went to a different site and the password was given to a buyer at a third site. Might that sufficiently confuse a freebie seeker?

              Just a stab in the dark here maybe KT knows better?
              I guess it's a possibility. I think that logically it makes sense. Anyone can respond to this?!
              Signature
              >> 12 Free Internet Marketing Tools You Should Be Using <<
              {{ DiscussionBoard.errors[1402758].message }}
              • Profile picture of the author ktmurf
                To be more clear, my receipt redirects are a third party site so it isn't exactly an issue for me.

                I don't generally put stuff like that in my robot.txt anyway because
                A. If it's not there to be found, it won't be found.
                B. There's little to no valuable content for a spider to crawl on that redirect anyway so even if it is indexed, the searches it will popup under are so obscure that no one will ever see it anyway.

                I'd imagine also leaving it out of your sitemap extension would be a good idea too.

                If you're getting good returns, you can still make individual logins/passwords by hand if it's worth your time to do that. It just takes a second, then you email it off to the buyer. Maybe 5 minutes of your time start to finish.
                {{ DiscussionBoard.errors[1403580].message }}
  • Profile picture of the author DogScout
    Robots Text is for saving the spider's time from crawling non-SEO info; like log-in pages, other sites on the same account, etc. It is NOT a protection device. In fact if used as such it is merely a map to information hackers might want, such as where you keep the goodies. Lol.
    Signature
    {{ DiscussionBoard.errors[1402736].message }}
  • Profile picture of the author samstephens
    Or if you're looking for a more automatic solution, you can check out DLGuard:

    DLGuard - Download page protector, create expiring download links

    This means you don't need to mess around with robots.txt files, you don't need to create passwords by hand, you don't need to create one general password (which isn't really a security feature - if you're going to just use one user/pass combo for everyone, then you're probably better off not using any at all).

    I know DLGuard costs money (simply because it costs me money to make, and LOTS of time/money to support), but the free solutions will also cost you time and money.

    By just implementing a decent sales/download/membership system, you'll save yourself a LOT of time and frustration in the long run.

    Let me know if you have any quesitons about how systems like DLGuard work, as I'm happy to help!

    cheers
    Sam
    Signature
    DLGuard v5 - The Warrior Edition
    Full integration with JVZoo, DigiResults, and WSO Pro for secure WSO's and WSO memberships.

    www.dlguard.com
    Serving the Warrior Forum since 2004
    {{ DiscussionBoard.errors[1404882].message }}
  • Profile picture of the author Thomas
    Here's an easier solution:

    1. Put the files you don't want people to access into a folder (on your website).
    2. Put an index page into the folder that redirects to yourdomain.com.
    3. Exclude the folder (not the specific file) in robots.txt.

    If you do that, anyone who opens the folder will be automatically redirected to the main page for your website. They won't be able to access a specific file unles they know the exact file name and location.

    That said, I'd recommend something like Sam's DLGuard too. It makes everything very easy and ensures nobody ever sees the exact location of the file to begin with.

    Regards,
    Tommy.
    {{ DiscussionBoard.errors[1404916].message }}

Trending Topics