Robots.txt advice

oliverjones · 08-31-2011, 03:12 PM

I have a site with loads of content relating to discussions etc. This is all good and indexed nicely by google. I also have 14k member profile pages with pretty much no content. I added these to my sitemap because I thought that the more pages we have, the bigger and better the engines will think we are. Is this the case or should I hide my profile pages using robots.txt as they contain very little content.

My other option would be to add links to the member pages with what they have written in various discussions but would this just be classed as duplicate content?

cassidywilliams · 08-31-2011, 03:17 PM

Google can take a sitemap with up to 50 000 URLs, so if your total amount of URLs is under that, its all good.

If your profile pages also link to your home page, more brownie points.

paulgl · 08-31-2011, 03:52 PM

Is this thread a joke? That's taking the sitemap concept from
benign to the ridiculous level.

Putting 14,000 profile pages in a sitemap is just insane.

Sometimes I think some of you sit on the john each morning
and come up with some crazy $#!t.

There is no brownie points for a sitemap. Where did you get
that idea?

Paul

jsmichael6 · 09-01-2011, 01:57 AM

"I added these to my sitemap because I thought that the more pages we have, the bigger and better the engines will think we are."

Where you read if you have more pages search engine will give you any advantage,this is not true.I dont think if 14k profile page will index then you will get any advantage.

StoneWilson · 09-01-2011, 02:28 AM

Quality is important, how these 14k "profile pages with pretty much no content" would help people? Think again.

08-31-2011, 03:12 PM	#1
oliverjones Oliver Join Date: Nov 2010 Posts: 143 Thanks: 1 Thanked 0 Times in 0 Posts	Robots.txt advice I have a site with loads of content relating to discussions etc. This is all good and indexed nicely by google. I also have 14k member profile pages with pretty much no content. I added these to my sitemap because I thought that the more pages we have, the bigger and better the engines will think we are. Is this the case or should I hide my profile pages using robots.txt as they contain very little content. My other option would be to add links to the member pages with what they have written in various discussions but would this just be classed as duplicate content?
	If you're looking to meet new people who are into the same stuff as you come and check out bongal.com

08-31-2011, 03:17 PM	#2
cassidywilliams HyperActive Warrior War Room Member Join Date: Feb 2008 Location: Austin, TX, USA. Posts: 210 Thanks: 4 Thanked 37 Times in 30 Posts	Re: Robots.txt advice Google can take a sitemap with up to 50 000 URLs, so if your total amount of URLs is under that, its all good. If your profile pages also link to your home page, more brownie points.
	»» Build better online courses and membership sites

08-31-2011, 03:52 PM	#3
paulgl Plundering the Web War Room Member Join Date: Feb 2007 Location: , , . Posts: 4,851 Thanks: 804 Thanked 1,200 Times in 887 Posts	Re: Robots.txt advice Is this thread a joke? That's taking the sitemap concept from benign to the ridiculous level. Putting 14,000 profile pages in a sitemap is just insane. Sometimes I think some of you sit on the john each morning and come up with some crazy $#!t. There is no brownie points for a sitemap. Where did you get that idea? Paul
	How to Make Money off Facebook: Login to your account. Deactivate your account. Get your butt to work.

09-01-2011, 01:57 AM	#4
jsmichael6 Active Warrior Join Date: Aug 2011 Posts: 45 Thanks: 0 Thanked 1 Time in 1 Post	Re: Robots.txt advice "I added these to my sitemap because I thought that the more pages we have, the bigger and better the engines will think we are." Where you read if you have more pages search engine will give you any advantage,this is not true.I dont think if 14k profile page will index then you will get any advantage.
	Coat stands UK \| a boards cheap

09-01-2011, 02:28 AM	#5
StoneWilson Advanced Warrior Join Date: Mar 2011 Posts: 693 Thanks: 69 Thanked 30 Times in 30 Posts	Re: Robots.txt advice Quality is important, how these 14k "profile pages with pretty much no content" would help people? Think again.
	Best Replica Watches\|Oakley眼镜官网\|charmzone