6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

Which files/folders should I Have Noindex in my Robots File?

by trentonlaura
5 replies
  • WEB DESIGN
    • |
I'm not sure what kinds of files and directories on my site I don't want crawled, right now I have everything open.

I know some webmasters have a specific /go/ directory for their affiliate links and they put that "go" directory as noindex in their robot file, I don't do this... could this have a negative impact on my site somehow?
#file #files or folders #noindex #robots
  • Profile picture of the author bestfriend
    I'm not sure what kinds of files and directories on my site I don't want crawled, right now I have everything open.
    If you don't have anything that you don't want to be indexed by SE bots, then it's okay leaving as is.

    could this have a negative impact on my site somehow?
    I don't think it will have any negative affect, because most probably there isn't any index file in the /go/ directory and if directory indexing off, search engines can't see anything by going there.
    {{ DiscussionBoard.errors[4420511].message }}
    • Profile picture of the author danilolee
      I usually block /checkout with robots. There's no need for such pages to show up on the SERPs
      {{ DiscussionBoard.errors[4420654].message }}
  • Profile picture of the author Kezz
    I'll make it easy for you.

    First, install the PC Robots.txt plugin which will get you half way there: WordPress › PC Robots.txt « WordPress Plugins

    Then also add the following:

    Disallow: /feed/
    Disallow: /comments/
    Disallow: /author/
    Disallow: /archives/
    Disallow: /20*
    Disallow: /trackback/
    There's a lot to understanding every element behind what to do, and it's not always a "one size fits all" deal.

    My partner wrote an in depth article explaining it all. Feel free to check it out: Using a Robots.txt File with WordPress | SpiderWeb Press
    {{ DiscussionBoard.errors[4422101].message }}
  • Profile picture of the author Snow_Predator
    Thank you Kezz. Your partner's article is truly outstanding. I left a comment to say thanks and I am taking his advice on the issue.
    {{ DiscussionBoard.errors[4474696].message }}

Trending Topics