Robots.txt file?

blalock61 · 11-20-2009, 02:41 PM

I need a little help with my Robots.txt file on one of my sites.

I have a site (I own the domain and have it on my own hosting) where on the front end I have uploaded a branded store from an affiliate network in the Health niche. I have also attached a blog to the backend of the site.

My question is how do I format the Robots.txt file to index the content on the backend blog and not the main domain.

The txt file that came with the branded store was as follows which disallows all bots from accessing any area of the store:

User-agent: *
Disallow: /

If I wanted the content on the backend blog indexed, would I format the file like this:

User-agent: *
Allow: /blog/
Disallow: /

Thank you for your help in this matter.

m4rx · 11-21-2009, 03:26 PM

I would say so.
I would probably do it like so though because the way you have it, everything will still be blocked.

User-agent: *
Disallow: /
Allow: /blog/

--m4rx

webtrix · 11-23-2009, 05:07 AM

Do you have TO MANY directories, like hundreds, or...?

Code:

User-agent: *
Allow: /blog/
Disallow: /directory1
Disallow: /directory2
Disallow: /directory3
Disallow: /directory4

vipdistinct · 11-23-2009, 11:00 PM

I would just add each page & folder as Disallow in the Robots but allow the blog. Also I had an idea for you on adding your pages and folders to the robots file. You should go to www.xml-sitemaps.com and generate a site map with them then download the url.txt list they give you and copy and paste those urls into a robots file.

11-20-2009, 02:41 PM	#1
blalock61 HyperActive Warrior War Room Member Join Date: Apr 2009 Location: GA,USA Posts: 152 Thanks: 152 Thanked 42 Times in 28 Posts Contact Info	Robots.txt file? I need a little help with my Robots.txt file on one of my sites. I have a site (I own the domain and have it on my own hosting) where on the front end I have uploaded a branded store from an affiliate network in the Health niche. I have also attached a blog to the backend of the site. My question is how do I format the Robots.txt file to index the content on the backend blog and not the main domain. The txt file that came with the branded store was as follows which disallows all bots from accessing any area of the store: User-agent: * Disallow: / If I wanted the content on the backend blog indexed, would I format the file like this: User-agent: * Allow: /blog/ Disallow: / Thank you for your help in this matter.

11-21-2009, 03:26 PM	#2
m4rx PHP Programmer Join Date: Nov 2009 Location: /dev/null Posts: 136 Thanks: 5 Thanked 12 Times in 12 Posts Social Networking Contact Info	Re: Robots.txt file? I would say so. I would probably do it like so though because the way you have it, everything will still be blocked. User-agent: * Disallow: / Allow: /blog/ --m4rx
	We are what we repeatedly do. Excellence, then, is not an act, but a HABIT. ~Aristotle Bored. Check out my blog.

11-23-2009, 05:07 AM	#3
webtrix HyperActive Warrior War Room Member Join Date: Oct 2009 Location: SE Europe Posts: 112 Thanks: 8 Thanked 8 Times in 8 Posts Social Networking Contact Info	Re: Robots.txt file? Do you have TO MANY directories, like hundreds, or...? Code: User-agent: * Allow: /blog/ Disallow: /directory1 Disallow: /directory2 Disallow: /directory3 Disallow: /directory4

11-23-2009, 11:00 PM	#4
vipdistinct Warrior Member Join Date: Aug 2009 Posts: 16 Thanks: 0 Thanked 2 Times in 2 Posts	Re: Robots.txt file? I would just add each page & folder as Disallow in the Robots but allow the blog. Also I had an idea for you on adding your pages and folders to the robots file. You should go to www.xml-sitemaps.com and generate a site map with them then download the url.txt list they give you and copy and paste those urls into a robots file.
	Travel Agent Atlanta