robots.txt vs. htaccess

by iamannie Posted: 14 years ago 9 replies

hello. i have a static website that is currently ranking for certain keywords. i decided to overhaul it to make it more seo friendly to boost up ranking. i am using wordpress as platform. the thing is, i would like to see it in beta first before actually "overwriting" the current content on my live site. therefore i created a /beta sub folder on my root. example: www(dot)mydomain(dot)com/beta. i DO NOT want the beta folder to be crawled by bots for fear it would hurt my current rankings. my questions:

1. was i on the right track on creating a /beta subfolder on the domain? i was thinking it was easier to launch the site once it is on the same domain. would it hurt my current rankings?
2. what are the rules in terms of robots.txt? there's no robots.txt on my root domain. do i create one now and add a disallow /beta tag on it? or do i create a robots.txt under /beta and make a disallow all?
3. do i make changes on htaccess as well? what do i specifically write on htaccess? which htaccess do i write to - root domain or /beta?
4. would adding meta noindex no follow on /beta header simply do the trick?
5. would disabling search engine crawling settings from wordpress dashboard/settings/privacy work?

i have a general knowledge of robots.txt and what it does but unsure how to go about this on a wordpress platform and on a domain that is currently ranking.

my only goals are for the search engines like google to NOT crawl this beta site and at the same time, NOT hurt my current rankings.

need your expert advice on this matter. thanks!

#search engine optimization #htaccess #robotstxt

dburk 14 years ago

Hi iamannie,

Just use the Privacy feature in Wordpress Settings to prevent search engines from spidering your beta site until you are ready to launch.
Matt Ward 14 years ago

Robots.txt is only a suggestion; no bot HAS to follow it.

You'd be much better off restricting it via htaccess if you really wanted to be secure.
- [2] replies
- AlbertSmiths 14 years ago
  
  Robots.txt - This tells the Search engines which URLs should not be indexed.
  
  .htaccess - This file can achieve many functions, such as
  
  - Ban a particular IP or User-agent.
  - Redirect URLs
  - Rewrite URLs with SE friendly names etc
  
  [1] reply
- strategic seo services 14 years ago
  
  I agree. htaccess is the way to go.
  
  [1] reply
smita 14 years ago

.htaccess is a configuration file which is used to restrict users from accessing your private pages. whereas Robots.txt is like a text file used by any websites owner to give to give instructions about their site to web robots.
chintangurjar 14 years ago

Just use the WordPress Privacy Settings to prevent search engines from spidering the site beta until you are ready to launch.
jonson 14 years ago

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

A .htaccess file is a directory-level configuration file supported by several web servers, that allows for decentralized management of web server configuration.The original purpose of .htaccess - reflected in its name was to allow per-directory access control, by for example requiring a password to access the content. Nowadays however, the .htaccess files can override many other configuration settings including content type and character set, CGI handlers, etc.

Join the discussion

Next Topics on Trending Feed

11

robots.txt vs. htaccess

iamannie • 14 years ago • 9 replies

hello. i have a static website that is currently ranking for certain keywords. i decided to overhaul it to make it more seo friendly to boost up ranking. i am using wordpress as platform. the thing is, i would like to see it in beta first before actually "overwriting" the current content on my live site. therefore i created a /beta sub folder on my root. example: www(dot)mydomain(dot)com/beta. i DO NOT want the beta folder to be crawled by bots for fear it would hurt my current rankings. my questions: 1. was i on the right track on creating a /beta subfolder on the domain? i was thinking it was easier to launch the site once it is on the same domain. would it hurt my current rankings?