How much important is robots.txt file in SEO

pjohn · 05-06-2011, 06:56 PM

Is it very important to have a robots.txt file for the SEO of your website? What are the benefits for having it and lose for not having it on our website?
Please explain. Thanks in anticipation.

Moneyerr · 05-08-2011, 05:23 AM

Robots.txt file allows or restricts to crawl some parts of your webpages. Search engines also find the location of the sitemap file in robots.txt which is very important in SEO view point. If you don't have robots.txt file, your server will return a 404 error page which can hurt your web rankings.

zoobie · 05-08-2011, 09:02 AM

Not important at all. Unless you want to block some part of your site from Search Engine access.

gittar1122 · 05-08-2011, 09:25 AM

Robots.txt tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. Keeping the search engines from accessing certain pages on your site is essential for privacy of your site.

Thanks

paulgl · 05-08-2011, 09:27 AM

404 errors happen all the time. They don't affect any rankings. Unless
you deleted a page by mistake. 404 pages are there for a reason. They
can make the visitor not leave your site, but find something else. A
404 error for an unneeded robots.txt is moot. 99.9999999% of normal
websites do not need a robots.txt

Webmasters who check stats just panic in seeing these errors. So
make one. Leave it blank. But then, what's the point?

Paul

omrishabbat · 05-08-2011, 09:29 AM

Creating a "good" Robots.txt is pretty basic but if you create it the wrong way it is crucial because you can say by mistake to search engines not to index your site or part of it!

App Developers · 05-08-2011, 11:09 AM

to my knowledge it is only used to block certain pages from the bots. I have never used it or needed to use it. I am not sure why you would want to block them.

aspnaren · 05-08-2011, 11:41 AM

robot.txt file contains the list of url that you dont want search engine bots t crawl from your sites..In some cases, for e.g, labels in blogger can be considered as copy content by SEs and you have more chance of getting punished by SEs...So its very important that you have a robots.txt if you label or categorize your content in your blogs

GuidoB · 05-08-2011, 12:52 PM

I don't think it really matters if you have one!

rijans · 05-08-2011, 09:12 PM

You can use robots.txt file to block duplicate links from search engines. So it's matter for SEO.

05-06-2011, 06:56 PM	#1
pjohn Active Warrior Join Date: Apr 2011 Posts: 43 Thanks: 0 Thanked 2 Times in 2 Posts	How much important is robots.txt file in SEO Is it very important to have a robots.txt file for the SEO of your website? What are the benefits for having it and lose for not having it on our website? Please explain. Thanks in anticipation.
	GPS Tracker \| Car GPS

05-08-2011, 05:23 AM	#2
Moneyerr HyperActive Warrior War Room Member Join Date: Jan 2011 Location: Living Free Posts: 325 Thanks: 0 Thanked 26 Times in 16 Posts	Re: How much important is robots.txt file in SEO Robots.txt file allows or restricts to crawl some parts of your webpages. Search engines also find the location of the sitemap file in robots.txt which is very important in SEO view point. If you don't have robots.txt file, your server will return a 404 error page which can hurt your web rankings.

05-08-2011, 09:02 AM	#3
zoobie Advanced Warrior War Room Member Join Date: Sep 2007 Location: hong kong Posts: 959 Blog Entries: 2 Thanks: 3 Thanked 42 Times in 37 Posts	Re: How much important is robots.txt file in SEO Not important at all. Unless you want to block some part of your site from Search Engine access.
	Quality Link Building - don't be the next victim of BuildMyRank.com - Check Out the Reviews Adsense Templates \|\| Free Keyword Tools \|\| SEO Hong Kong \|\| Relieve Stress \|\| Buy Solar Charger

05-08-2011, 09:25 AM	#4
gittar1122 Advanced Warrior Join Date: Sep 2010 Location: Internet Posts: 555 Thanks: 10 Thanked 26 Times in 25 Posts Contact Info	Re: How much important is robots.txt file in SEO Robots.txt tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. Keeping the search engines from accessing certain pages on your site is essential for privacy of your site. Thanks
	50% Discount For Next 9 Members High Quality Dofollow Blog Commenting Services Quality Dofollow Backlinks High PR Link Building

05-08-2011, 09:27 AM	#5
paulgl Plundering the Web War Room Member Join Date: Feb 2007 Location: , , . Posts: 4,849 Thanks: 804 Thanked 1,199 Times in 886 Posts	Re: How much important is robots.txt file in SEO 404 errors happen all the time. They don't affect any rankings. Unless you deleted a page by mistake. 404 pages are there for a reason. They can make the visitor not leave your site, but find something else. A 404 error for an unneeded robots.txt is moot. 99.9999999% of normal websites do not need a robots.txt Webmasters who check stats just panic in seeing these errors. So make one. Leave it blank. But then, what's the point? Paul
	How to Make Money off Facebook: Login to your account. Deactivate your account. Get your butt to work.

05-08-2011, 09:29 AM	#6
omrishabbat HyperActive Warrior Join Date: May 2010 Posts: 219 Thanks: 0 Thanked 20 Times in 16 Posts Social Networking	Re: How much important is robots.txt file in SEO Creating a "good" Robots.txt is pretty basic but if you create it the wrong way it is crucial because you can say by mistake to search engines not to index your site or part of it!

05-08-2011, 11:09 AM	#7
App Developers HyperActive Warrior Join Date: Feb 2011 Location: San Antonio, Texas Posts: 446 Thanks: 71 Thanked 16 Times in 16 Posts	Re: How much important is robots.txt file in SEO to my knowledge it is only used to block certain pages from the bots. I have never used it or needed to use it. I am not sure why you would want to block them.
	Baby Monitors Pilot Shops How to get girls

05-08-2011, 11:41 AM	#8
aspnaren Warrior Member Join Date: Feb 2011 Posts: 22 Thanks: 0 Thanked 1 Time in 1 Post	Re: How much important is robots.txt file in SEO robot.txt file contains the list of url that you dont want search engine bots t crawl from your sites..In some cases, for e.g, labels in blogger can be considered as copy content by SEs and you have more chance of getting punished by SEs...So its very important that you have a robots.txt if you label or categorize your content in your blogs
	Facebook tips and tricks

05-08-2011, 12:52 PM	#9
GuidoB Warrior Member Join Date: May 2011 Posts: 19 Thanks: 0 Thanked 1 Time in 1 Post	Re: How much important is robots.txt file in SEO I don't think it really matters if you have one!
	Be a Guest Writer on my Blog with your links !Click Here! or Read about it here (Main Page is PR6)

05-08-2011, 09:12 PM	#10
rijans Active Warrior Join Date: May 2011 Posts: 36 Thanks: 3 Thanked 1 Time in 1 Post Contact Info	Re: How much important is robots.txt file in SEO You can use robots.txt file to block duplicate links from search engines. So it's matter for SEO.