11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

robots.txt vs. htaccess

by iamannie
9 replies
  • SEO
    • |
hello. i have a static website that is currently ranking for certain keywords. i decided to overhaul it to make it more seo friendly to boost up ranking. i am using wordpress as platform. the thing is, i would like to see it in beta first before actually "overwriting" the current content on my live site. therefore i created a /beta sub folder on my root. example: www(dot)mydomain(dot)com/beta. i DO NOT want the beta folder to be crawled by bots for fear it would hurt my current rankings. my questions:

1. was i on the right track on creating a /beta subfolder on the domain? i was thinking it was easier to launch the site once it is on the same domain. would it hurt my current rankings?
2. what are the rules in terms of robots.txt? there's no robots.txt on my root domain. do i create one now and add a disallow /beta tag on it? or do i create a robots.txt under /beta and make a disallow all?
3. do i make changes on htaccess as well? what do i specifically write on htaccess? which htaccess do i write to - root domain or /beta?
4. would adding meta noindex no follow on /beta header simply do the trick?
5. would disabling search engine crawling settings from wordpress dashboard/settings/privacy work?

i have a general knowledge of robots.txt and what it does but unsure how to go about this on a wordpress platform and on a domain that is currently ranking.

my only goals are for the search engines like google to NOT crawl this beta site and at the same time, NOT hurt my current rankings.

need your expert advice on this matter. thanks!
#htaccess #robotstxt
  • Profile picture of the author dburk
    Hi iamannie,

    Just use the Privacy feature in Wordpress Settings to prevent search engines from spidering your beta site until you are ready to launch.
    Signature

    * Get Results - Outsource Your PPC Management |
    *     Don Burk Advertising & Marketing - www.donburk.com

    {{ DiscussionBoard.errors[5465865].message }}
  • Profile picture of the author Matt Ward
    Robots.txt is only a suggestion; no bot HAS to follow it.

    You'd be much better off restricting it via htaccess if you really wanted to be secure.
    Signature
    "Keep moving forward."
    {{ DiscussionBoard.errors[5465871].message }}
  • Profile picture of the author smita
    .htaccess is a configuration file which is used to restrict users from accessing your private pages. whereas Robots.txt is like a text file used by any websites owner to give to give instructions about their site to web robots.
    {{ DiscussionBoard.errors[5537163].message }}
  • Profile picture of the author chintangurjar
    Just use the WordPress Privacy Settings to prevent search engines from spidering the site beta until you are ready to launch.
    {{ DiscussionBoard.errors[5537178].message }}
  • Profile picture of the author jonson
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

    A .htaccess file is a directory-level configuration file supported by several web servers, that allows for decentralized management of web server configuration.The original purpose of .htaccess - reflected in its name was to allow per-directory access control, by for example requiring a password to access the content. Nowadays however, the .htaccess files can override many other configuration settings including content type and character set, CGI handlers, etc.
    {{ DiscussionBoard.errors[5537269].message }}

Trending Topics