Robots.txt?

jesicawillss · 12-20-2010, 05:43 AM

What is robots.txt? Is it a part of on-page?

marketgoal · 12-20-2010, 05:56 AM

Robots.txt is also the factor of On page optimization. Basically Robots.txt is a text file you put on your website to tell search robots which pages you would like them not to visit. Here in this text file you can define that pages which you don't want to crawl by the crawler.

Devkumar · 12-20-2010, 06:46 AM

Robots.txt is a text file where we can block the spider from calling some important files.

paulgl · 12-20-2010, 06:52 AM

A normal website does not really need a robots.txt. The search bots will
automatically look for one first, and if not found, will give a not found error.
No big deal. Using robots.txt to block some pages is the correct way of doing
it. There is no need to block anything, unless you have sensitive material
readily found. There is no need to block privacy policy, etc., as google is
smart enough to know what to do with those pages. There is no danger
in having a privacy policy being indexed or showing up in SERPs (unlikely
anyway, and no big deal).

If you don't like seeing errors, just create a robots.txt and leave it blank.

Paul

12-20-2010, 05:43 AM	#1
jesicawillss Jesica Wills Join Date: Nov 2010 Location: India Posts: 45 Thanks: 0 Thanked 0 Times in 0 Posts	Robots.txt? What is robots.txt? Is it a part of on-page?

12-20-2010, 05:56 AM	#2
marketgoal seojoy Join Date: Nov 2010 Location: www.topseo4indiavg.com Posts: 39 Thanks: 0 Thanked 0 Times in 0 Posts Social Networking	Re: Robots.txt? Robots.txt is also the factor of On page optimization. Basically Robots.txt is a text file you put on your website to tell search robots which pages you would like them not to visit. Here in this text file you can define that pages which you don't want to crawl by the crawler.
	Free Samples

12-20-2010, 06:46 AM	#3
Devkumar Warrior Member Join Date: Dec 2010 Posts: 3 Thanks: 0 Thanked 0 Times in 0 Posts	Re: Robots.txt? Robots.txt is a text file where we can block the spider from calling some important files.

12-20-2010, 06:52 AM	#4
paulgl Plundering the Web War Room Member Join Date: Feb 2007 Location: , , . Posts: 4,849 Thanks: 804 Thanked 1,199 Times in 886 Posts	Re: Robots.txt? A normal website does not really need a robots.txt. The search bots will automatically look for one first, and if not found, will give a not found error. No big deal. Using robots.txt to block some pages is the correct way of doing it. There is no need to block anything, unless you have sensitive material readily found. There is no need to block privacy policy, etc., as google is smart enough to know what to do with those pages. There is no danger in having a privacy policy being indexed or showing up in SERPs (unlikely anyway, and no big deal). If you don't like seeing errors, just create a robots.txt and leave it blank. Paul
	How to Make Money off Facebook: Login to your account. Deactivate your account. Get your butt to work.