What does the symbol * mean in robots.txt?

1 replies
  • SEO
  • |
I've been trying to optimize my robots.txt file to get rid of the crawl errors, but I am wondering what the symbol "*" means in the robots.txt file?

For example, what is the difference between the three below:

Disallow: /category/
Disallow: /category/*/*
Disallow: /category/*/*/*

Does it mean not to crawl any further pages or files in sub-directories? Just confused, your help would be greatly appreciated!
#robotstxt #symbol
  • Profile picture of the author jsmichael6
    Its wild card character.Yes it mean not to crawl any further pages or files in sub-directories
    {{ DiscussionBoard.errors[4580922].message }}

Trending Topics