web crawlers and root directory

Alimoqi · 03-03-2011, 09:10 PM

Hi everyone,
Does google index pages that are in the root directory on the server but not linked to from any of the pages on the website? I work with another guy on a few sites and we have some test pages in the root directory (so we can both access them) that are basically duplicates of some of the other pages and I don't want them crawled. Is this a problem or should I put them in a separate folder that is blocked by robots.txt?
Thanks!

parbat · 03-03-2011, 10:37 PM

no. Big G Bot visits pages from one to another. if you havnt putted your link of that page anywhere on your site then Bot will not reach that page. I Belive that Bot always start with index files.

i have several other files like yahoo site validation file, bing files and some other files which has no relation with my public pages.If you using wordpress you can hide your perticular page from search engine visibility.

den1231 · 03-04-2011, 01:04 AM

On my side of idea, I guess the bot will still crawled the links which are in the root, as long as they are properly categorized for each page.

03-03-2011, 09:10 PM	#1
Alimoqi Warrior Member Join Date: Feb 2011 Posts: 18 Thanks: 0 Thanked 0 Times in 0 Posts	web crawlers and root directory Hi everyone, Does google index pages that are in the root directory on the server but not linked to from any of the pages on the website? I work with another guy on a few sites and we have some test pages in the root directory (so we can both access them) that are basically duplicates of some of the other pages and I don't want them crawled. Is this a problem or should I put them in a separate folder that is blocked by robots.txt? Thanks!

03-03-2011, 10:37 PM	#2
parbat crazzy warrior Join Date: Mar 2011 Posts: 59 Thanks: 1 Thanked 8 Times in 8 Posts Contact Info	Re: web crawlers and root directory no. Big G Bot visits pages from one to another. if you havnt putted your link of that page anywhere on your site then Bot will not reach that page. I Belive that Bot always start with index files. i have several other files like yahoo site validation file, bing files and some other files which has no relation with my public pages.If you using wordpress you can hide your perticular page from search engine visibility.
	developers forum \| upsc syllabus \| Treatment for acne \| pinoy chat

03-04-2011, 01:04 AM	#3
den1231 SEO Join Date: Mar 2011 Posts: 27 Thanks: 0 Thanked 1 Time in 1 Post	Re: web crawlers and root directory On my side of idea, I guess the bot will still crawled the links which are in the root, as long as they are properly categorized for each page.