Robots.txt directory and file question

by Ralph Moore

Posted: 12 years ago 8 replies

WEB DESIGN

Most of the definitive guides to the robots.txt file are not, so I have a question.

If I do the following:

User-agent: *
Disallow: /templates/

does this tell the bots to ignore everything in just the templates directory or does it also tell them to disallow any sub-directories as well?

such as /templates/images/

And does anyone have a link to a really good robots.txt guide?

Thanks.

#directory #file #question #robotstxt

xtrapunch 12 years ago

The above code will ask search bots to ignore the entire folder and its contents.
- Thanks
- 1 reply
Signature

>> Web Design, Wordpress & SEO - XtraPunch.com <<
Web Design & SEO Agency | Serving World Wide from New Delhi, India
{{ DiscussionBoard.errors[5309370].message }}
- Ralph Moore 12 years ago
  
  Originally Posted by xtrapunch
  
  The above code will ask search bots to ignore the entire folder and its contents.
  
  Thanks. That is what I thought, but I was not sure.
  
  Thanks
  
  Signature
  
  Work From Home Reports
  
  {{ DiscussionBoard.errors[5311876].message }}
RobKonrad 12 years ago

Hey there,

did you check

The Web Robots Pages

Cheers,
Rob
- Thanks
- 1 reply
Signature

================================================== ===
This blog is awesome: http://www.robkonrad.com/blog. Read it.
================================================== ===
{{ DiscussionBoard.errors[5309642].message }}
- Ralph Moore 12 years ago
  
  Originally Posted by RobKonrad
  
  Hey there,
  
  did you check
  
  The Web Robots Pages
  
  Cheers,
  Rob
  
  Thanks guys, I did in fact take a look at that, but honestly, I was not impressed as it leaves a LOT of questions unanswered.
  
  I have yet to see a really good explanation of all that the robots.txt file can do on a granular level.
  
  And I am pretty sure it can do a lot more than people think it can.
  
  Thanks
  
  Signature
  
  Work From Home Reports
  
  {{ DiscussionBoard.errors[5311891].message }}
kokopelli 12 years ago

Try: The Web Robots Pages
- Thanks
Signature
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{{ DiscussionBoard.errors[5309879].message }}
NetMan 12 years ago

Or this cool robot.txt generator

Robot.Txt Generator

Regards,

Andre Foisy
- Thanks
{{ DiscussionBoard.errors[5312163].message }}
watson88 12 years ago

Try to remove last / from your robot.txt file.
- Thanks
{{ DiscussionBoard.errors[5313559].message }}
Venus Brown 12 years ago

It'll ignore all the folder content including its files.
- Thanks
Signature
Social Network Developers
{{ DiscussionBoard.errors[5313799].message }}

Robots.txt directory and file question

Trending Topics

Boat and Jet Ski Rentals...... Door Hangers Effective?

A Very Strange Internet Problem

Are billboards still effective in driving customers?

Let's dissect the genius behind Nike's 'Just Do It' slogan. What makes it so powerful and timeless?

Google Ads, landing pages, and Clickbank links