How to not let bots index some pages

[iceq]

Hi there,

I have a small website with 8 pages, and 4 of them (including index.html)
ar the same.

How to let bots index only the index.html page, for not considering other
pages duplicated content?


Thanks!

[aron111]

If you don't want the spiders to visit some of your pages, you can stop them from crawling the pages by uploading robots.txt in the root folder, by using robots you can guide the spiders to visit the pages.

if you want to block the spiders from crawling all your website then you should write the robot as:
User-Agent: *
Disallow: /

'*'- indicates the spiders, you can change it accordingly,

if you want to block a particular page, you can use the format::

User-Agent: *
Disallow: /yourpage.extension

and for multiple pages you should write it separately as:

User-Agent: *
Disallow: /yourpage1.extension
Disallow: /yourpage2.extension
Disallow: /yourpage3.extension
Disallow: /yourpage4.extension

....etc

Regards,

[iceq]

Thanks guys