How do i set some content not to be crawled/indexed?

2 replies
I have some pages with a lot of duplicate content. Like 20 pages where over half the content is duplicate on each page. I want to tell the search engines NOT to index/crawl this part of the page, is this possible?

The site makes me money, but recently had a huge drop in traffic because of this problem i'm very sure.

Thanks
#content #crawled or indexed #set
  • Profile picture of the author JayXtreme
    Matt

    You need to specify this in a robots.txt file..

    INsert it into your robots.txt file like this:

    User-Agent: *
    Disallow: /name of folder/

    Then include any and all folder/pages that you don't want to include in this fashion.

    The [ * ] above is a wildcard so will apply to all search engines, but you can specify each bot by name if you like.. remember, the MSNBot won't index your content the same as Google/Yahoo.. so it is sometimes worth having different rules for different bots.. if you are using a strategy for the MSN search engine etc.


    Peace

    Jay
    Signature

    Bare Murkage.........

    {{ DiscussionBoard.errors[523581].message }}
  • Profile picture of the author Matt Hoey
    The thing is though it's not certain pages i want to not be crawled, it's parts of the content on a page.

    ...or am i missing something?
    {{ DiscussionBoard.errors[523591].message }}

Trending Topics