8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

google is indexing my sitemap

by lunchboxdiet
7 replies
hi,

google is indexing my sitemap.xml on one of my sites and i wondered how to stop it to prevent people finding my product pages etc- if i put something in robots.txt (not sure what) then will that stop google from indexing properly.

Please help.

Regards

Simon
#google #indexing #sitemap
  • Profile picture of the author smseleem
    Yes you can use the robots.txt file to stop certain pages or folders from being crawled by robots or search engines.
    {{ DiscussionBoard.errors[5686575].message }}
  • Profile picture of the author kokopelli
    That's the whole idea ... that the search engines should index your sitemap.xml file. If you do not want pages indexed, either add the necessary meta tags, or exclude them via a robots.txt file. You can also password-protect them, or their directories.
    Signature
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    {{ DiscussionBoard.errors[5686658].message }}
  • Profile picture of the author udikafri
    here is an example of using the robot.txt (also for disallowing indexing) :

    http://www.robotstxt.org/orig.html
    {{ DiscussionBoard.errors[5696825].message }}
  • Profile picture of the author pbarnhart
    no, you don't use the robots.txt file for this.

    Chances are, you have a link someplace to your sitemap.xml file. The proper response is to issue a X-Robots-Tag: noindex directive for it - I have this in my .htaccess file for all .xml files:
    Code:
    <filesMatch ".(xml)$">
X-Robots-Tag: noindex
</filesMatch>
    Signature
    Connect up! I work for Starbucks Lattes
    {{ DiscussionBoard.errors[5697920].message }}
  • Profile picture of the author onzevil
    It is learned that a lot of good in it.
    {{ DiscussionBoard.errors[5698235].message }}
  • Profile picture of the author wordcatcher
    The robots.txt protocol is widely adopted by web spiders and crawlers. You need a robots.txt file only if your site includes content that you don't want search engines to index.
    {{ DiscussionBoard.errors[5808072].message }}
  • Profile picture of the author Earnie Boyd
    The sitemap.xml file is also an open specification sitemaps.org - Protocol. It should not contain links that you do not want anonymous users finding. How is sitemap.xml created? Is it hand edited or CMS driven? If CMS driven then you should ask for support in you CMS support area.
    Signature
    Earnie Boyd
    {{ DiscussionBoard.errors[5809505].message }}

Trending Topics