What does the symbol * mean in robots.txt?

futurepocket · 08-30-2011, 03:51 PM

I've been trying to optimize my robots.txt file to get rid of the crawl errors, but I am wondering what the symbol "*" means in the robots.txt file?

For example, what is the difference between the three below:

Disallow: /category/
Disallow: /category/*/*
Disallow: /category/*/*/*

Does it mean not to crawl any further pages or files in sub-directories? Just confused, your help would be greatly appreciated!

jsmichael6 · 08-31-2011, 07:32 AM

Its wild card character.Yes it mean not to crawl any further pages or files in sub-directories

08-30-2011, 03:51 PM	#1
futurepocket HyperActive Warrior Join Date: Aug 2011 Posts: 121 Thanks: 3 Thanked 0 Times in 0 Posts	*What does the symbol mean in robots.txt?** I've been trying to optimize my robots.txt file to get rid of the crawl errors, but I am wondering what the symbol "" means in the robots.txt file? For example, what is the difference between the three below: Disallow: /category/ Disallow: /category//* Disallow: /category///* Does it mean not to crawl any further pages or files in sub-directories? Just confused, your help would be greatly appreciated!

08-31-2011, 07:32 AM	#2
jsmichael6 Active Warrior Join Date: Aug 2011 Posts: 45 Thanks: 0 Thanked 1 Time in 1 Post	*Re: What does the symbol mean in robots.txt?** Its wild card character.Yes it mean not to crawl any further pages or files in sub-directories