Robots.txt analysis tool interpretation?
Hello,
This doesn't make any sense to me and I cannot figure our if my robots.txt file is working the way I want it to or not?
The goal here is to have everything in those folders (go, goto, and visit) to be not crawled and indexed by google or any other search bots.
/go/********.html
Robots allowed: All robots
Robots disallowed: All robots (line 4: Disallow: /go/)
/go/******.html
Robots allowed: All robots
Robots disallowed: All robots (line 4: Disallow: /go/)
/goto/********.html
Robots allowed: All robots
Robots disallowed: All robots (line 5: Disallow: /goto/)
/goto/*********.html
Robots allowed: All robots
Robots disallowed: All robots (line 5: Disallow: /goto/)
/visit/******.html
Robots allowed: All robots
Robots disallowed: All robots (line 3: Disallow: /visit/)
/visit/******-*****.html
Robots allowed: All robots
Robots disallowed: All robots (line 3: Disallow: /visit/)
Thanks for any help here!
This doesn't make any sense to me and I cannot figure our if my robots.txt file is working the way I want it to or not?
The goal here is to have everything in those folders (go, goto, and visit) to be not crawled and indexed by google or any other search bots.
/go/********.html
Robots allowed: All robots
Robots disallowed: All robots (line 4: Disallow: /go/)
/go/******.html
Robots allowed: All robots
Robots disallowed: All robots (line 4: Disallow: /go/)
/goto/********.html
Robots allowed: All robots
Robots disallowed: All robots (line 5: Disallow: /goto/)
/goto/*********.html
Robots allowed: All robots
Robots disallowed: All robots (line 5: Disallow: /goto/)
/visit/******.html
Robots allowed: All robots
Robots disallowed: All robots (line 3: Disallow: /visit/)
/visit/******-*****.html
Robots allowed: All robots
Robots disallowed: All robots (line 3: Disallow: /visit/)
Thanks for any help here!
-
Mark Andrews Banned-
Thanks
{{ DiscussionBoard.errors[6236594].message }} -