Go Back   WarriorForum - Internet Marketing Forums > The Warrior Forum > Main Internet Marketing Discussion Forum
Register Blogs FAQ Social Groups CalendarHelp Desk

Reply
 
LinkBack Thread Tools
Old 05-13-2009, 11:33 AM   #1
Marketer / Writer
 
The Lady's Avatar
 
Join Date: Oct 2007
Location: South Central USA.
Posts: 365
Blog Entries: 10
Thanks: 127
Thanked 3 Times in 3 Posts
Social Networking View Member's FaceBook Profile  View Member's Twitter Profile 
Contact Info
Send a message via Skype™ to The Lady
Default Quick Question: Robot.Txt

I did a very quick search. I didn't see this aspect addressed, but I may have missed it honestly.

I'm about to launch a new set of programs, but I want to set it up so that the crawl doesn't include files loaded on the site. I know I can set up a robot instruction to follow certain paths on the site, but how would I do it to exclude files/zip'd or otherwise? TIA

The Lady is offline   Reply With Quote
Old 05-13-2009, 02:08 PM   #2
Marketer / Writer
 
The Lady's Avatar
 
Join Date: Oct 2007
Location: South Central USA.
Posts: 365
Blog Entries: 10
Thanks: 127
Thanked 3 Times in 3 Posts
Social Networking View Member's FaceBook Profile  View Member's Twitter Profile 
Contact Info
Send a message via Skype™ to The Lady
Default Re: Quick Question: Robot.Txt

do i just 'don't follow' .zip? ect?

The Lady is offline   Reply With Quote
Old 05-13-2009, 02:16 PM   #3
Senior Warrior Member
War Room Member
 
GeorgR.'s Avatar
 
Join Date: Jun 2007
Location: Earth
Posts: 5,945
Blog Entries: 15
Thanks: 72
Thanked 870 Times in 578 Posts
Social Networking View Member's Twitter Profile 
Default Re: Quick Question: Robot.Txt

User-agent: *
Disallow: /

DISALLOWS ALL FILES/FOLDERS

-------------------------------------------------------

User-agent: *
Disallow: /somefolder/

DISALLOWS CRAWLING FOR "SOMEFOLDER"


---------------------------------------------------------

User-agent: *
Disallow: /aaa/
Disallow: /bbb/

DISALLOWS CRAWLING OF aaa and bbb


Uhm...maybe you can move all ZIP files into a folder "zip" and disallow the zip folder?

Cheers

*** Affiliate Site Quick --> The Fastest & Easiest Way to Make Affiliate Sites!<--
-> VISIT www.1UP-SEO.com *** <- Internet Marketing, SEO Tips, Reviews & More!! ***
Free Ipad * LinkAloha * Kindle Vs Ipad * What is the Best Tablet
Article Spinning Service - The BEST SPUN ARTICLES you Have Ever Seen! - Thread Here *
GeorgR. is online now   Reply With Quote
Old 05-13-2009, 02:49 PM   #4
Socrates: I drank What?
War Room Member
 
jacktackett's Avatar
 
Join Date: May 2007
Location: Cary,NC , USA.
Posts: 1,378
Blog Entries: 1
Thanks: 777
Thanked 450 Times in 253 Posts
Social Networking View Member's FaceBook Profile  View Member's Twitter Profile 
Contact Info
Send a message via Skype™ to jacktackett
Default Re: Quick Question: Robot.Txt

Check out this post a while back:

Robot Text?

hope this helps,
best,
--Jack

Mega Monster WSO for KimW
http://ow.ly/4JdHm
Help Keep Ken Strong!
http://goo.gl/Q0c99
4th Annual Warrior Event! March,2012

jacktackett is offline   Reply With Quote
Old 05-13-2009, 03:28 PM   #5
Active Warrior
War Room Member
 
Join Date: Jan 2009
Posts: 77
Thanks: 10
Thanked 5 Times in 5 Posts
Default Re: Quick Question: Robot.Txt

Disallow a folder, disallow indexing of the folder and name the file something obscure or else people could simply check your robots.txt file and download your product for free.
Gimme4Free is offline   Reply With Quote
Reply

  WarriorForum - Internet Marketing Forums > The Warrior Forum > Main Internet Marketing Discussion Forum

Tags
question, quick, robottxt

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off



All times are GMT -6. The time now is 11:42 PM.