Using Noindex within a Sitemap?

by 8 replies
10
#search engine optimization #noindex #sitemap
  • That would be a highly non-standard usage. I recommend your robots.txt usage only uses what is accepted standard.
    • [ 1 ] Thanks
    • [1] reply
    • You are both in error.

      Anyway, like all big sites that have searches, reservations,
      or anything like that, you don't WANT to index that stuff.

      If by chance the googlebot visits when a search, reservation,
      etc. is made, then those results could be indexed. And, in
      some strange cases, show up in SERPS and would be pretty
      lame looking, if not generate a ton of error messages if actually
      clicked on.

      So, they don't want them indexed.

      Common practice and is very standard usage.

      Paul
      • [2] replies
  • Actually there is such a directive as "noindex" in the robots.txt. Although not used frequently, it is a valid reference that some bots comply with.

    "Disallowed" pages can actually STILL be indexed in some cases. When that does happen, they appear with no information, just the Url in the title of the SERP pages.

    So that's why they have that directive. They probably had issues with Google still indexing pages which they have disallowed.

    If your CMS allows, it's probably easier to manage this stuff using a meta tag for your various types of content.
    • [ 1 ] Thanks
    • [2] replies
    • Most any CMS worth it's weight already has a robots.txt in place,
      exactly the way it should be.

      Getting back to my point, hotels.com, etc. do not
      want spur of the moment internet bargains that go away
      to the next visitor to be indexed.

      The OP is a tad mixed up on sitemaps and robots.txt.

      Paul
      • [ 1 ] Thanks
    • Thanks a lot!

      I searched on the web and haven't found anything on this topic, can you please give me some references?

      Why does that happen? So adding a 'noindex' in page level will prevent from this confusion? I mean, will adding a 'noindex' meta tags in pages that I don't want to be indexed will make sure that they are not indexed?

      Yes that's a valid point, but Google says it only recognizes 'Disallow' and 'Allow' in a robots.txt?!

      Once again thanks!
  • @All

    I am sorry for a wrong Title to my post guys. It has to read 'Using Noindex within a Robots.txt?
    Thanks!
Join the discussion

Next Topics on Trending Feed