Robots.txt file

Robots.txt files inform search engine spiders how to interact with indexing your content.

* By default search engines are greedy. They want to index as much high quality information as they can, and they will assume that they can crawl everything unless you tell them otherwise.
* If you specify data for all bots (*) and data for a specific bot (like GoogleBot) then the specific bot commands will be followed while that engine ignores the global/default bot commands.

The following pages are likely to be in robots.txt

1. Login page
2. 404 error page
3. any other page that you don't want to be get indexed.

Internal search pages can be blocked from search engines to be crawled. You can use this coding in robots.txt to block search pages.


Disallow: /*?

Actually, the best practice is to not have one.

But, since bots look for them, they will return an error
if you look at your web stats. The error is moot. But
webmasters do not like errors, so they panic and think
they need one. If so, just leave it blank. This is 2012.
There really is no need to block anything from a normal
site. Google is smart enough to know what and when to
show it in SERPs.

The exception would be if you have some type of a search
feature that searches your site and gives posted results.
Then, if you had boatloads of people doing this, these
results may get indexed needlessly and just be useless bloat.

Rarely does one actually need a robots.txt with anything in it.

For fun, I just checked the warriorforum's. They don't even block
search results. They don't block anything except that sometimes
pesky custom crawler from 80legs.

Since the WF blocks nothing from google, I'd take that as a good
example for what I was talking about.

Paul

Hi In my Opinion mostly Logon page use every websites user.

disallow all secret folders and files. Thats it done.

disallow folder containing important files which you do not want google to crawl and show when someone searches on web. like thank you page, etc.

According to me basically set the robots.txt

User-agent: *
Disallow:
Sitemap: http://www.xyz.com/sitemap.xml

If you dont want search engines to crawl any web page, then you can use robot.txt on that page code. There is no disadvantage from this code.

You shall disallow any content which you donot want to show to your website visitor such as your admin area files/folders.