Quick Question: Robot.Txt

The Lady · 05-13-2009, 11:33 AM

I did a very quick search. I didn't see this aspect addressed, but I may have missed it honestly.

I'm about to launch a new set of programs, but I want to set it up so that the crawl doesn't include files loaded on the site. I know I can set up a robot instruction to follow certain paths on the site, but how would I do it to exclude files/zip'd or otherwise? TIA

The Lady · 05-13-2009, 02:08 PM

do i just 'don't follow' .zip? ect?

GeorgR. · 05-13-2009, 02:16 PM

User-agent: *
Disallow: /

DISALLOWS ALL FILES/FOLDERS

-------------------------------------------------------

User-agent: *
Disallow: /somefolder/

DISALLOWS CRAWLING FOR "SOMEFOLDER"

---------------------------------------------------------

User-agent: *
Disallow: /aaa/
Disallow: /bbb/

DISALLOWS CRAWLING OF aaa and bbb

Uhm...maybe you can move all ZIP files into a folder "zip" and disallow the zip folder?

Cheers

jacktackett · 05-13-2009, 02:49 PM

Check out this post a while back:

Robot Text?

hope this helps,
best,
--Jack

Gimme4Free · 05-13-2009, 03:28 PM

Disallow a folder, disallow indexing of the folder and name the file something obscure or else people could simply check your robots.txt file and download your product for free.

05-13-2009, 11:33 AM	#1
The Lady Marketer / Writer Join Date: Oct 2007 Location: South Central USA. Posts: 365 Blog Entries: 10 Thanks: 127 Thanked 3 Times in 3 Posts Social Networking Contact Info	Quick Question: Robot.Txt I did a very quick search. I didn't see this aspect addressed, but I may have missed it honestly. I'm about to launch a new set of programs, but I want to set it up so that the crawl doesn't include files loaded on the site. I know I can set up a robot instruction to follow certain paths on the site, but how would I do it to exclude files/zip'd or otherwise? TIA
	~Catrina~ Original Content Ready for Sale New Writers * ProductivePLR * Submit A Job

05-13-2009, 02:08 PM	#2
The Lady Marketer / Writer Join Date: Oct 2007 Location: South Central USA. Posts: 365 Blog Entries: 10 Thanks: 127 Thanked 3 Times in 3 Posts Social Networking Contact Info	Re: Quick Question: Robot.Txt do i just 'don't follow' .zip? ect?
	~Catrina~ Original Content Ready for Sale New Writers * ProductivePLR * Submit A Job

05-13-2009, 02:16 PM	#3
GeorgR. Senior Warrior Member War Room Member Join Date: Jun 2007 Location: Earth Posts: 5,945 Blog Entries: 15 Thanks: 72 Thanked 870 Times in 578 Posts Social Networking	Re: Quick Question: Robot.Txt User-agent: * Disallow: / DISALLOWS ALL FILES/FOLDERS ------------------------------------------------------- User-agent: * Disallow: /somefolder/ DISALLOWS CRAWLING FOR "SOMEFOLDER" --------------------------------------------------------- User-agent: * Disallow: /aaa/ Disallow: /bbb/ DISALLOWS CRAWLING OF aaa and bbb Uhm...maybe you can move all ZIP files into a folder "zip" and disallow the zip folder? Cheers
	* Affiliate Site Quick --> The Fastest & Easiest Way to Make Affiliate Sites!<-- -> VISIT www.1UP-SEO.com * <- Internet Marketing, SEO Tips, Reviews & More!! * Free Ipad** * LinkAloha * Kindle Vs Ipad * What is the Best Tablet Article Spinning Service - The BEST SPUN ARTICLES you Have Ever Seen! - Thread Here *

05-13-2009, 02:49 PM	#4
jacktackett Socrates: I drank What? War Room Member Join Date: May 2007 Location: Cary,NC , USA. Posts: 1,378 Blog Entries: 1 Thanks: 777 Thanked 450 Times in 253 Posts Social Networking Contact Info	Re: Quick Question: Robot.Txt Check out this post a while back: Robot Text? hope this helps, best, --Jack
	Mega Monster WSO for KimW http://ow.ly/4JdHm Help Keep Ken Strong! http://goo.gl/Q0c99 4th Annual Warrior Event! March,2012

05-13-2009, 03:28 PM	#5
Gimme4Free Active Warrior War Room Member Join Date: Jan 2009 Posts: 77 Thanks: 10 Thanked 5 Times in 5 Posts	Re: Quick Question: Robot.Txt Disallow a folder, disallow indexing of the folder and name the file something obscure or else people could simply check your robots.txt file and download your product for free.