3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

web crawlers and root directory

by Alimoqi
2 replies
  • SEO
    • |
Hi everyone,
Does google index pages that are in the root directory on the server but not linked to from any of the pages on the website? I work with another guy on a few sites and we have some test pages in the root directory (so we can both access them) that are basically duplicates of some of the other pages and I don't want them crawled. Is this a problem or should I put them in a separate folder that is blocked by robots.txt?
Thanks!
#crawlers #directory #root #web
  • Profile picture of the author parbat
    no. Big G Bot visits pages from one to another. if you havnt putted your link of that page anywhere on your site then Bot will not reach that page. I Belive that Bot always start with index files.

    i have several other files like yahoo site validation file, bing files and some other files which has no relation with my public pages.If you using wordpress you can hide your perticular page from search engine visibility.
    {{ DiscussionBoard.errors[3469189].message }}
    • Profile picture of the author den1231
      On my side of idea, I guess the bot will still crawled the links which are in the root, as long as they are properly categorized for each page.
      {{ DiscussionBoard.errors[3469637].message }}

Trending Topics