How to not let bots index some pages

by 2 replies
3
Hi there,

I have a small website with 8 pages, and 4 of them (including index.html)
ar the same.

How to let bots index only the index.html page, for not considering other
pages duplicated content?


Thanks!
#search engine optimization #bots #index #pages
  • If you don't want the spiders to visit some of your pages, you can stop them from crawling the pages by uploading robots.txt in the root folder, by using robots you can guide the spiders to visit the pages.

    if you want to block the spiders from crawling all your website then you should write the robot as:
    User-Agent: *
    Disallow: /

    '*'- indicates the spiders, you can change it accordingly,

    if you want to block a particular page, you can use the format::

    User-Agent: *
    Disallow: /yourpage.extension

    and for multiple pages you should write it separately as:

    User-Agent: *
    Disallow: /yourpage1.extension
    Disallow: /yourpage2.extension
    Disallow: /yourpage3.extension
    Disallow: /yourpage4.extension

    ....etc

    Regards,
  • Banned
    [DELETED]
  • Thanks guys
Join the discussion

Next Topics on Trending Feed