4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

Robots.txt file?

by blalock61
3 replies
I need a little help with my Robots.txt file on one of my sites.

I have a site (I own the domain and have it on my own hosting) where on the front end I have uploaded a branded store from an affiliate network in the Health niche. I have also attached a blog to the backend of the site.

My question is how do I format the Robots.txt file to index the content on the backend blog and not the main domain.

The txt file that came with the branded store was as follows which disallows all bots from accessing any area of the store:

User-agent: *
Disallow: /

If I wanted the content on the backend blog indexed, would I format the file like this:

User-agent: *
Allow: /blog/
Disallow: /

Thank you for your help in this matter.
#file #robotstxt
  • Profile picture of the author m4rx
    I would say so.
    I would probably do it like so though because the way you have it, everything will still be blocked.

    User-agent: *
    Disallow: /
    Allow: /blog/

    --m4rx
    Signature
    We are what we repeatedly do. Excellence, then, is not an act, but a HABIT. ~Aristotle
    Bored. Learn everything you need to know about Organic Gardening.
    {{ DiscussionBoard.errors[1412329].message }}
  • Profile picture of the author webtrix
    Do you have TO MANY directories, like hundreds, or...?

    Code:
    User-agent: *
Allow: /blog/
Disallow: /directory1
Disallow: /directory2
Disallow: /directory3
Disallow: /directory4
    {{ DiscussionBoard.errors[1417542].message }}
  • Profile picture of the author vipdistinct
    I would just add each page & folder as Disallow in the Robots but allow the blog. Also I had an idea for you on adding your pages and folders to the robots file. You should go to www.xml-sitemaps.com and generate a site map with them then download the url.txt list they give you and copy and paste those urls into a robots file.
    {{ DiscussionBoard.errors[1420539].message }}

Trending Topics