web crawlers and root directory

by Alimoqi 2 replies
Hi everyone,
Does google index pages that are in the root directory on the server but not linked to from any of the pages on the website? I work with another guy on a few sites and we have some test pages in the root directory (so we can both access them) that are basically duplicates of some of the other pages and I don't want them crawled. Is this a problem or should I put them in a separate folder that is blocked by robots.txt?
Thanks!
#search engine optimization #crawlers #directory #root #web
  • Profile picture of the author parbat
    no. Big G Bot visits pages from one to another. if you havnt putted your link of that page anywhere on your site then Bot will not reach that page. I Belive that Bot always start with index files.

    i have several other files like yahoo site validation file, bing files and some other files which has no relation with my public pages.If you using wordpress you can hide your perticular page from search engine visibility.
    {{ DiscussionBoard.errors[3469189].message }}
    • Profile picture of the author den1231
      On my side of idea, I guess the bot will still crawled the links which are in the root, as long as they are properly categorized for each page.
      {{ DiscussionBoard.errors[3469637].message }}

Trending Topics

  • 96 {{ upvoteCount | shortNum }}

    Warriors .... Warriors.... what would you like to LEARN?

    Good Morning awesome Warriors... I'm happy to announce that The Warrior Forum is planning to create our first COURSE. Yes... Powered by Warrior Wisdom... yeah!! Made at home. But before ... [read more]

  • 53 {{ upvoteCount | shortNum }}

    How I Sold 985 Copies of an Affiliate Offer in 5 Days (Step by Step)

    When I first started online in 2002, it took me a year to sell 985 copies of anything. These days, I can typically do it in the span of a ... [read more]

  • 22 {{ upvoteCount | shortNum }}

    1 Powerful Concept that Grows Your Business

    Connect with people interested in what you have to offer. Mindfully. Versus desperately running from prospect to prospect, or from number to number. This idea sounds simple, and like common ... [read more]

  • 13 {{ upvoteCount | shortNum }}

    Can we trust Review Copy reviews of WSO offers?

    Hello all I have seen in various WSO threads, users asking for a review copy of the offer. Usually, the user then posts a positive review of the WSO for ... [read more]

  • 1 {{ upvoteCount | shortNum }}

    Link social on Backpage .com?

    Aknness2016 in Social Networks

    How do I link social media when posting on Backpage.com?