Can robots.txt do this?

by 4 replies
5
Can a robots.txt file do all this?

1 Leave main page indexed
2 Remove all other pages in main folder
3 Leave a sub-folder and all its child folders indexed
#website design #robotstxt
  • Yes, it can.

    That file is intended for just that purpose - instruct robots/spiders to crawl or ignore files or directories. You will need to write the specific instructions for the robots in that file.

    If you have never written a robots.txt file before - you may want to start by looking at a few instructions online.

    How to Write a robots.txt File
    The Web Robots Pages
    http://www.mcanerin.com/EN/search-engine/robots-txt.asp

    Just a personal note : Do you have a good reason for blocking access to so much of your site? This type of action could hinder any organic SEO effort.
    • [ 2 ] Thanks
    • [1] reply
    • Thanks. Yes I suspect my customer is going to want to redirect the current homepage to the new version of the site I built for him on a sub-domain and I want to prevent duplicate content penalties.At some point I'll move it all anyway.
  • Are you using WP??
    • [1] reply
    • The new site (currently on the sub-domain) is WordPress. The original site was built in Joomla.
Join the discussion

Next Topics on Trending Feed