How to use a robots.txt file?

[Kanute]

Is it important to use robots.txt files on your website? What is it for, and how is it used? Should a robots.txt file be added to every page?

[adrenalinfeed]

No. You don´t need to use one. But you could just write one default, as all robots visiting your site will search for it (basically to see where they are allowed to go, and where they should go).

You will only need a robots.txt file in your root folder. A robots.txt could look like this:

User-agent: *
Disallow: /i-want-this-to-be-unindexed
Sitemap: url-to-sitemap/sitemap.xml

This one says:
1. The star indicates that the robots.txt is talking to every robot.
2. Dissallowed areas to go on the server.
3. Where is the sitemap located.

[CoverGraphix]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

In the old days it was more useful than today, but it is still useful especially if you want to exclude certain pages from being indexed.

[ashleysmith12]

Hi,

Robots.txt file basically used to allow or disallow the directoiries by google crawler which is to index or which is not to be index.

[Adam Smith]

When would you want to use the disallow? I thought that you would want every part of your site indexed?

[brunom]

Originally Posted by bestiphone5cases

When would you want to use the disallow? I thought that you would want every part of your site indexed?

If you're selling a product, you wouldn't want your thank you page indexed, now would you?

[Chris Cole]

Originally Posted by bestiphone5cases

When would you want to use the disallow? I thought that you would want every part of your site indexed?

If you have download pages, or seperate pages unlinked or standalone that you do not want everybody to see on your website.

That is of course, if you do not have some software/script protecting the your content on your sites. Then you'd want to leave pages un indexed.

[CoverGraphix]

Originally Posted by bestiphone5cases

When would you want to use the disallow? I thought that you would want every part of your site indexed?

Download pages, thankyou pages, members only areas that are not secured, all of those you would disallow.

But, another time to do it is when you would have duplicate content, say you have a post and you have an archive. Google doesn't want to index the same post twice, so it is best to disallow the archive so you don't double post to google.

[Adam Smith]

Ahhh all makes sense now Thanks for the insight guys

[Kanute]

Thanks a lot for helpful information on this topic!

[Stuart william]

Originally Posted by adrenalinfeed

No. You don´t need to use one. But you could just write one default, as all robots visiting your site will search for it (basically to see where they are allowed to go, and where they should go).

You will only need a robots.txt file in your root folder. A robots.txt could look like this:

User-agent: *
Disallow: /i-want-this-to-be-unindexed
Sitemap: url-to-sitemap/sitemap.xml

This one says:
1. The star indicates that the robots.txt is talking to every robot.
2. Dissallowed areas to go on the server.
3. Where is the sitemap located.

you can create and upload it on your root folder