How to Protect Robots.txt File

wealthydon · 06-07-2009, 07:54 PM

Hi Warriors,

I am curious if it is possible to
protect robots.txt file that is
uploaded on the server root
directory.

I noticed that it is easy for smart
marketer to look up the robots.txt
file and "see behind the hood."

I know of DLGuard, et al but I
am thinking of a FREE METHOD
instead.

Thank you.

BJ Min · 06-07-2009, 07:57 PM

how do you use dlguard with it?

Ryan_Taylor · 06-07-2009, 07:58 PM

I didn't think you could because it has to be publicly available.

Craig Desorcy · 06-07-2009, 07:59 PM

Protecting your robot txt is useless. It's there to communicate with the search engines.
What's there to protect?

Craig

AlphaWarrior · 06-07-2009, 08:45 PM

I do not know what you are trying to protect, but here goes one possibility. In the robots.txt file, a viewer can see what you do not want a spider to index. If you list single files, viewers can see what you do not want indexed and look at the files anyway. However, if you block a subdirectory rather than block a single file, the viewer will not see what files are in the subdirectory and, therefore, cannot look at the files.

DrewG · 06-07-2009, 09:17 PM

Try this instead maybe -

Code:

<html>
<head>
 <meta name="robots" content="noindex" />
 <title>Don't index this page</title>
</head>

in your HTML code of your site, that way you don't need a robots.txt file,

I'd also like someone to shed some light on this...

p2y · 06-07-2009, 10:24 PM

It's impossible to have a functional robots.txt file unless you're doing IP based cloaking, in which case sooner or later you'd probably get caught by google and banned

samstephens · 06-09-2009, 02:59 AM

It's a common mistake to put secure folders into robots.txt files, thinking you're making them secure from search engines (someone must be teaching this as a security method, but it's just plain wrong).

As wealthydon said, anyone can just read them.

They're simply not designed to stop search engines from indexing your download pages.

Drew has listed another method to stop pages being listed in search engines, but keep in mind not all search engines use these rules.

There isn't a free way that is also secure, simply because the technology is quite involved to make a decent download protection system.

I know you said you didn't want to invest in DLGuard, but perhaps after you make a few sales you could simply roll that money back into your business, and invest in it. That means no out of pocket money, and it will also increase security as well as ease of use for your business.

I hope this helps!

In short: never put download links in your robots.txt files!

cheers
Sam

06-07-2009, 07:54 PM	#1
wealthydon HyperActive Warrior Join Date: Sep 2008 Posts: 154 Thanks: 2 Thanked 12 Times in 7 Posts	How to Protect Robots.txt File Hi Warriors, I am curious if it is possible to protect robots.txt file that is uploaded on the server root directory. I noticed that it is easy for smart marketer to look up the robots.txt file and "see behind the hood." I know of DLGuard, et al but I am thinking of a FREE METHOD instead. Thank you.
	Free Affiliate Marketing Guide \| Cheap Article Writers \| Flip Websites for Profit \| Work from Home Affiliate Marketing \| Free PLR Articles and Ebooks \| Best Treatment for Acne \| Bacterial Vaginosis Natural Treatment

06-07-2009, 07:57 PM	#2
BJ Min Senior Warrior Member War Room Member Join Date: Jun 2007 Location: Los Angeles, CA Posts: 1,675 Thanks: 148 Thanked 36 Times in 19 Posts Social Networking Contact Info	Re: How to Protect Robots.txt File how do you use dlguard with it?
	Looking to hire Freelance Writers (NEWBIES welcome)...

06-07-2009, 07:58 PM	#3
Ryan_Taylor Advanced Warrior War Room Member Join Date: Feb 2008 Location: Chicago, IL Posts: 660 Thanks: 10 Thanked 17 Times in 9 Posts	Re: How to Protect Robots.txt File I didn't think you could because it has to be publicly available.

06-07-2009, 07:59 PM	#4
Craig Desorcy Senior Warrior Member War Room Member Join Date: Jan 2004 Location: Japan Posts: 2,376 Blog Entries: 1 Thanks: 418 Thanked 913 Times in 211 Posts Social Networking	Re: How to Protect Robots.txt File Protecting your robot txt is useless. It's there to communicate with the search engines. What's there to protect? Craig
	Make Money Writing \| Internet Business Coach \| How to start a blog \| High Quality PLR

06-07-2009, 08:45 PM	#5
AlphaWarrior HyperActive Warrior Join Date: Nov 2008 Posts: 144 Thanks: 4 Thanked 11 Times in 9 Posts	Re: How to Protect Robots.txt File I do not know what you are trying to protect, but here goes one possibility. In the robots.txt file, a viewer can see what you do not want a spider to index. If you list single files, viewers can see what you do not want indexed and look at the files anyway. However, if you block a subdirectory rather than block a single file, the viewer will not see what files are in the subdirectory and, therefore, cannot look at the files.

06-07-2009, 09:17 PM	#6
DrewG UltimateIMGraphics.com War Room Member Join Date: Feb 2009 Posts: 539 Blog Entries: 9 Thanks: 113 Thanked 77 Times in 59 Posts	Re: How to Protect Robots.txt File Try this instead maybe - Code: <html> <head> <meta name="robots" content="noindex" /> <title>Don't index this page</title> </head> in your HTML code of your site, that way you don't need a robots.txt file, I'd also like someone to shed some light on this...
	--> 100 Social Bookmarks for just $5 - Check Out My Fiverr Gig <-- SocialMonkee.com - GET 25 FREE Social Bookmarks PER DAY - COMPLETELY AUTOMATED Web 2.0 Graphics \| Internet Marketing Graphics \| Ecover Action Scripts

06-07-2009, 10:24 PM	#7
p2y Advanced Warrior War Room Member Join Date: Feb 2007 Posts: 508 Thanks: 9 Thanked 187 Times in 57 Posts	Re: How to Protect Robots.txt File It's impossible to have a functional robots.txt file unless you're doing IP based cloaking, in which case sooner or later you'd probably get caught by google and banned
	Automatic Profile Backlinks - Free Demo

06-09-2009, 02:59 AM	#8
samstephens DLGuard.com War Room Member Join Date: Nov 2004 Location: Central Coast, Australia Posts: 6,199 Thanks: 67 Thanked 257 Times in 117 Posts Social Networking	Re: How to Protect Robots.txt File It's a common mistake to put secure folders into robots.txt files, thinking you're making them secure from search engines (someone must be teaching this as a security method, but it's just plain wrong). As wealthydon said, anyone can just read them. They're simply not designed to stop search engines from indexing your download pages. Drew has listed another method to stop pages being listed in search engines, but keep in mind not all search engines use these rules. There isn't a free way that is also secure, simply because the technology is quite involved to make a decent download protection system. I know you said you didn't want to invest in DLGuard, but perhaps after you make a few sales you could simply roll that money back into your business, and invest in it. That means no out of pocket money, and it will also increase security as well as ease of use for your business. I hope this helps! In short: never put download links in your robots.txt files! cheers Sam
	DLGuard v4.6 Now Released! Full integration with Amazon S3 for authenticated large file and high bandwidth downloads. www.dlguard.com Where size doesn't matter!