Keeping search bots out--pls double check my robots.txt file...

Alan Petersen · 03-20-2009, 03:43 PM

Hello there,

I want to keep Google and any other search bots out of one of my directories. My understanding is I can do this by creating a robots.txt file. So I uploaded it to public_html...is that the correct spot to have it in the root? And here is the code...does it look okay?

Code:

User-Agent: *
Disallow: /folder_name

I get paranoid that I'll keep Google out of the whole site. I just want them out of this one directory only.

Thanks!

MemberWing · 03-20-2009, 03:48 PM

To be sure - signup for Google webmasters tools and it has robots.txt checking facility there.

Alan Petersen · 03-20-2009, 03:56 PM

Thanks for the tip. I'm trying to verify the site but they're unable to verify due to this error:

Quote:

We've detected that your 404 (file not found) error page returns a status of 200 (Success) in the header.

Not finding on the answer on how to fix it from google, any suggestions?

Alan Petersen · 03-20-2009, 04:11 PM

Quote:

Originally Posted by K.Saravana Kumar

Looks like you are using mod_rewrite or .htaccess modified. you need to fix that return 404 error code for unfound pages .

Cool thanks! I had this in my .htaccess file:

Code:

ErrorDocument 404 /likes/index.php

So it was going back to my index page vs. displaying a 404 error page. I think I read that this was a good thing to do so I put it in there so folks wouldn't get a 404 page. But I took it out and now it works! And google has verified my site. All just edit the 404 page with an ad or something. :-)

Thanks!

tgrpublishing · 03-21-2009, 04:59 AM

Just a note - robots.txt doesn't GUARANTEE that bots won't access the directory. Only ethically written bots will avoid it. But, that does mean at least the big 3 will

ehicks727 · 03-21-2009, 11:01 AM

Quote:

Originally Posted by TigerPublishing

Just a note - robots.txt doesn't GUARANTEE that bots won't access the directory. Only ethically written bots will avoid it. But, that does mean at least the big 3 will

Yeah, that's the plain truth. I wrote a custom analytics program, and I have to maintain a list of over 1000 bots now so I can screen out non-human traffic.

The worst is the scrapers who just insert random text into the browser user-agent.

Don't think that putting something in your robots.txt will keep all the bots out. Most bots roaming the Internet aren't respectful.

03-20-2009, 03:43 PM	#1
Alan Petersen Senior Warrior Member War Room Member Join Date: Oct 2006 Location: San Francisco, CA Posts: 3,386 Blog Entries: 11 Thanks: 723 Thanked 778 Times in 494 Posts Social Networking	Keeping search bots out--pls double check my robots.txt file... Hello there, I want to keep Google and any other search bots out of one of my directories. My understanding is I can do this by creating a robots.txt file. So I uploaded it to public_html...is that the correct spot to have it in the root? And here is the code...does it look okay? Code: User-Agent: * Disallow: /folder_name I get paranoid that I'll keep Google out of the whole site. I just want them out of this one directory only. Thanks!
	New WSO: Cash In With Amazon's Forgotten Business! How To Turn Friendly Conversations into Cash Internet Marketing Blog \| \|Join the Amazon Self Publisher's Group \| eBay Affiliates Only

03-20-2009, 03:48 PM	#2
MemberWing Gleb War Room Member Join Date: Dec 2008 Location: Ottawa, Canada Posts: 845 Thanks: 10 Thanked 104 Times in 85 Posts Social Networking Contact Info	Re: Keeping search bots out--pls double check my robots.txt file... To be sure - signup for Google webmasters tools and it has robots.txt checking facility there.
	Wordpress Membership Plugin MemberWing-X Build automated recurring subscription website with Wordpress Membership plugin MemberWing \| Law of Attraction Coaching \| Love and Relationships Forum \| Law of Attraction

03-21-2009, 04:59 AM	#5
tgrpublishing Warrior Member Join Date: Feb 2009 Posts: 26 Thanks: 0 Thanked 7 Times in 5 Posts	Re: Keeping search bots out--pls double check my robots.txt file... Just a note - robots.txt doesn't GUARANTEE that bots won't access the directory. Only ethically written bots will avoid it. But, that does mean at least the big 3 will