Robots, Wordpress and Duplicate Content

Beldin · 11-30-2009, 11:17 AM

Hi gang,

Wordpress and all its subdirectories are new to me.... Is there a standard robots.txt people use when they use wordpress so that only the pages and posts are indexed?

So far I've got:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
But think I should also have

Disallow: /tag/
Disallow: /category/
Disallow: /author/
Disallow: /admin/

And something for archives... /archives/? Maybe?

Any suggestions?

Thanks,
Stephen

Intrepreneur · 11-30-2009, 01:41 PM

Let everything be indexed even your contact page, privacy policy etc.

Having too many restrictions can affect your site performance.

Cristian Lungu · 12-01-2009, 12:04 PM

I should restrain SE bot crawl pages like terms of service and privacy policy and about us, since this dilutes the link juice passed between your important web pages. Just a suggestion...

If you're new to wordpress just use All in One SEO pack plugin and your SEO will be taken care of; as your duplicate content issue.

tony-raymondo · 12-01-2009, 12:44 PM

Quote:

Originally Posted by Beldin

Hi gang,

Wordpress and all its subdirectories are new to me.... Is there a standard robots.txt people use when they use wordpress so that only the pages and posts are indexed?

So far I've got:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
But think I should also have

Disallow: /tag/
Disallow: /category/
Disallow: /author/
Disallow: /admin/

And something for archives... /archives/? Maybe?

Any suggestions?

Thanks,
Stephen

why would you want to?

lots of those are important

Beldin · 12-02-2009, 12:57 PM

I have no reason too... just searching the web and found some suggestions... so I came here to verify!

11-30-2009, 11:17 AM	#1
Beldin Active Warrior War Room Member Join Date: Nov 2009 Posts: 79 Thanks: 3 Thanked 2 Times in 2 Posts	Robots, Wordpress and Duplicate Content Hi gang, Wordpress and all its subdirectories are new to me.... Is there a standard robots.txt people use when they use wordpress so that only the pages and posts are indexed? So far I've got: User-agent: * Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /cgi-bin/ But think I should also have Disallow: /tag/ Disallow: /category/ Disallow: /author/ Disallow: /admin/ And something for archives... /archives/? Maybe? Any suggestions? Thanks, Stephen

11-30-2009, 01:41 PM	#2
Intrepreneur I.C.Hope War Room Member Join Date: Apr 2009 Location: Northern Ireland Posts: 2,515 Thanks: 446 Thanked 227 Times in 181 Posts Contact Info	Re: Robots, Wordpress and Duplicate Content Let everything be indexed even your contact page, privacy policy etc. Having too many restrictions can affect your site performance.
	I want a good keyword researcher, not for min sites but for tech articles. Hit me up if you've got those skillz!

12-01-2009, 12:04 PM	#3
Cristian Lungu Cristian Lungu Join Date: Sep 2009 Location: Romania Posts: 24 Thanks: 0 Thanked 1 Time in 1 Post Social Networking	Re: Robots, Wordpress and Duplicate Content I should restrain SE bot crawl pages like terms of service and privacy policy and about us, since this dilutes the link juice passed between your important web pages. Just a suggestion... If you're new to wordpress just use All in One SEO pack plugin and your SEO will be taken care of; as your duplicate content issue.
	TopNotch Tips for Generating Traffic to Your Online Business - back on my blog

12-02-2009, 12:57 PM	#5
Beldin Active Warrior War Room Member Join Date: Nov 2009 Posts: 79 Thanks: 3 Thanked 2 Times in 2 Posts	Re: Robots, Wordpress and Duplicate Content I have no reason too... just searching the web and found some suggestions... so I came here to verify!