robots.txt exclusion question

2 replies
Is it possible for the robots.txt file to exclude robots from the top level directory, but still allow them access to subdirectories?
#exclusion #question #robotstxt
  • Profile picture of the author silverskin
    I don't think it work...

    User-agent: *
    Allow: /private/
    Allow: /images-saved/
    Allow: /images-working/
    Disallow: /
    {{ DiscussionBoard.errors[2115675].message }}
  • Profile picture of the author webtrix
    Or something like this:

    Disallow: /*.php$
    Disallow: /*.htm$
    {{ DiscussionBoard.errors[2119793].message }}

Trending Topics