Robots.txt file?

by 3 replies
4
I need a little help with my Robots.txt file on one of my sites.

I have a site (I own the domain and have it on my own hosting) where on the front end I have uploaded a branded store from an affiliate network in the Health niche. I have also attached a blog to the backend of the site.

My question is how do I format the Robots.txt file to index the content on the backend blog and not the main domain.

The txt file that came with the branded store was as follows which disallows all bots from accessing any area of the store:

User-agent: *
Disallow: /

If I wanted the content on the backend blog indexed, would I format the file like this:

User-agent: *
Allow: /blog/
Disallow: /

Thank you for your help in this matter.
#programming #file #robotstxt
  • I would say so.
    I would probably do it like so though because the way you have it, everything will still be blocked.

    User-agent: *
    Disallow: /
    Allow: /blog/

    --m4rx
    • [ 1 ] Thanks
  • Do you have TO MANY directories, like hundreds, or...?

    Code:
    User-agent: *
Allow: /blog/
Disallow: /directory1
Disallow: /directory2
Disallow: /directory3
Disallow: /directory4
    • [ 1 ] Thanks
  • I would just add each page & folder as Disallow in the Robots but allow the blog. Also I had an idea for you on adding your pages and folders to the robots file. You should go to www.xml-sitemaps.com and generate a site map with them then download the url.txt list they give you and copy and paste those urls into a robots file.
    • [ 1 ] Thanks
Join the discussion

Next Topics on Trending Feed

  • 4

    Robots.txt file?

    I need a little help with my Robots.txt file on one of my sites. I have a site (I own the domain and have it on my own hosting) where on the front end I have uploaded a branded store from an affiliate network in the Health niche. I have also attached a blog to the backend of the site.