11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

How to Protect Robots.txt File

by wealthydon
7 replies
Hi Warriors,

I am curious if it is possible to
protect robots.txt file that is
uploaded on the server root
directory.

I noticed that it is easy for smart
marketer to look up the robots.txt
file and "see behind the hood."

I know of DLGuard, et al but I
am thinking of a FREE METHOD
instead.

Thank you.
#file #protect #robotstxt
  • Profile picture of the author BJ Min
    how do you use dlguard with it?
    Signature
    Interviews with Internet Marketing Millionaires! | YouTube Video Marketing | Video Marketing Tips
    How to Publish Your Amazon Ebook + Paperback + Audibook & Make Passive Income Online! |
    {{ DiscussionBoard.errors[855815].message }}
  • Profile picture of the author Ryan_Taylor
    I didn't think you could because it has to be publicly available.
    Signature

    {{ DiscussionBoard.errors[855817].message }}
  • Profile picture of the author AlphaWarrior
    I do not know what you are trying to protect, but here goes one possibility. In the robots.txt file, a viewer can see what you do not want a spider to index. If you list single files, viewers can see what you do not want indexed and look at the files anyway. However, if you block a subdirectory rather than block a single file, the viewer will not see what files are in the subdirectory and, therefore, cannot look at the files.
    {{ DiscussionBoard.errors[855897].message }}
  • Profile picture of the author p2y
    It's impossible to have a functional robots.txt file unless you're doing IP based cloaking, in which case sooner or later you'd probably get caught by google and banned
    Signature
    Automatic Profile Backlinks - Free Demo
    {{ DiscussionBoard.errors[856134].message }}
  • Profile picture of the author samstephens
    It's a common mistake to put secure folders into robots.txt files, thinking you're making them secure from search engines (someone must be teaching this as a security method, but it's just plain wrong).

    As wealthydon said, anyone can just read them.

    They're simply not designed to stop search engines from indexing your download pages.

    Drew has listed another method to stop pages being listed in search engines, but keep in mind not all search engines use these rules.

    There isn't a free way that is also secure, simply because the technology is quite involved to make a decent download protection system.

    I know you said you didn't want to invest in DLGuard, but perhaps after you make a few sales you could simply roll that money back into your business, and invest in it. That means no out of pocket money, and it will also increase security as well as ease of use for your business.

    I hope this helps!


    In short: never put download links in your robots.txt files!

    cheers
    Sam
    Signature
    DLGuard v5 - The Warrior Edition
    Full integration with JVZoo, DigiResults, and WSO Pro for secure WSO's and WSO memberships.

    www.dlguard.com
    Serving the Warrior Forum since 2004
    {{ DiscussionBoard.errors[860130].message }}

Trending Topics