3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

How to not let bots index some pages

by iceq
2 replies
  • SEO
    • |
Hi there,

I have a small website with 8 pages, and 4 of them (including index.html)
ar the same.

How to let bots index only the index.html page, for not considering other
pages duplicated content?


Thanks!
#bots #index #pages
  • Profile picture of the author aron111
    If you don't want the spiders to visit some of your pages, you can stop them from crawling the pages by uploading robots.txt in the root folder, by using robots you can guide the spiders to visit the pages.

    if you want to block the spiders from crawling all your website then you should write the robot as:
    User-Agent: *
    Disallow: /

    '*'- indicates the spiders, you can change it accordingly,

    if you want to block a particular page, you can use the format::

    User-Agent: *
    Disallow: /yourpage.extension

    and for multiple pages you should write it separately as:

    User-Agent: *
    Disallow: /yourpage1.extension
    Disallow: /yourpage2.extension
    Disallow: /yourpage3.extension
    Disallow: /yourpage4.extension

    ....etc

    Regards,
    {{ DiscussionBoard.errors[1822744].message }}
  • Profile picture of the author iceq
    Thanks guys
    {{ DiscussionBoard.errors[1822762].message }}

Trending Topics