robots.txt vs. htaccess

iamannie · 01-21-2012, 02:43 PM

hello. i have a static website that is currently ranking for certain keywords. i decided to overhaul it to make it more seo friendly to boost up ranking. i am using wordpress as platform. the thing is, i would like to see it in beta first before actually "overwriting" the current content on my live site. therefore i created a /beta sub folder on my root. example: www(dot)mydomain(dot)com/beta. i DO NOT want the beta folder to be crawled by bots for fear it would hurt my current rankings. my questions:

1. was i on the right track on creating a /beta subfolder on the domain? i was thinking it was easier to launch the site once it is on the same domain. would it hurt my current rankings?
2. what are the rules in terms of robots.txt? there's no robots.txt on my root domain. do i create one now and add a disallow /beta tag on it? or do i create a robots.txt under /beta and make a disallow all?
3. do i make changes on htaccess as well? what do i specifically write on htaccess? which htaccess do i write to - root domain or /beta?
4. would adding meta noindex no follow on /beta header simply do the trick?
5. would disabling search engine crawling settings from wordpress dashboard/settings/privacy work?

i have a general knowledge of robots.txt and what it does but unsure how to go about this on a wordpress platform and on a domain that is currently ranking.

my only goals are for the search engines like google to NOT crawl this beta site and at the same time, NOT hurt my current rankings.

need your expert advice on this matter. thanks!

dburk · 01-21-2012, 07:04 PM

Hi iamannie,

Just use the Privacy feature in Wordpress Settings to prevent search engines from spidering your beta site until you are ready to launch.

Matt Ward · 01-21-2012, 07:05 PM

Robots.txt is only a suggestion; no bot HAS to follow it.

You'd be much better off restricting it via htaccess if you really wanted to be secure.

AlbertSmiths · 01-21-2012, 11:42 PM

Robots.txt - This tells the Search engines which URLs should not be indexed.

.htaccess - This file can achieve many functions, such as

- Ban a particular IP or User-agent.
- Redirect URLs
- Rewrite URLs with SE friendly names etc

iamannie · 01-31-2012, 03:12 PM

thanks for the response guys!

strategic seo services · 01-31-2012, 03:34 PM

Quote:

Originally Posted by Matt Ward

Robots.txt is only a suggestion; no bot HAS to follow it.

You'd be much better off restricting it via htaccess if you really wanted to be secure.

I agree. htaccess is the way to go.

paulgl · 01-31-2012, 05:15 PM

The answer is simpler than that. Just password protect
the folder. Done and done. Very easy if you have cpanel.

Robots.txt is iffy. Google is not the only place that indexes
sites.

In reality, there is no reason to NOT play around with it,
live.

Paul

smita · 02-02-2012, 02:37 AM

.htaccess is a configuration file which is used to restrict users from accessing your private pages. whereas Robots.txt is like a text file used by any websites owner to give to give instructions about their site to web robots.

chintangurjar · 02-02-2012, 02:41 AM

Just use the WordPress Privacy Settings to prevent search engines from spidering the site beta until you are ready to launch.

jonson · 02-02-2012, 03:03 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

A .htaccess file is a directory-level configuration file supported by several web servers, that allows for decentralized management of web server configuration.The original purpose of .htaccess - reflected in its name was to allow per-directory access control, by for example requiring a password to access the content. Nowadays however, the .htaccess files can override many other configuration settings including content type and character set, CGI handlers, etc.

01-21-2012, 02:43 PM	#1
iamannie Warrior Member Join Date: Jan 2012 Posts: 3 Thanks: 0 Thanked 0 Times in 0 Posts	robots.txt vs. htaccess hello. i have a static website that is currently ranking for certain keywords. i decided to overhaul it to make it more seo friendly to boost up ranking. i am using wordpress as platform. the thing is, i would like to see it in beta first before actually "overwriting" the current content on my live site. therefore i created a /beta sub folder on my root. example: www(dot)mydomain(dot)com/beta. i DO NOT want the beta folder to be crawled by bots for fear it would hurt my current rankings. my questions: 1. was i on the right track on creating a /beta subfolder on the domain? i was thinking it was easier to launch the site once it is on the same domain. would it hurt my current rankings? 2. what are the rules in terms of robots.txt? there's no robots.txt on my root domain. do i create one now and add a disallow /beta tag on it? or do i create a robots.txt under /beta and make a disallow all? 3. do i make changes on htaccess as well? what do i specifically write on htaccess? which htaccess do i write to - root domain or /beta? 4. would adding meta noindex no follow on /beta header simply do the trick? 5. would disabling search engine crawling settings from wordpress dashboard/settings/privacy work? i have a general knowledge of robots.txt and what it does but unsure how to go about this on a wordpress platform and on a domain that is currently ranking. my only goals are for the search engines like google to NOT crawl this beta site and at the same time, NOT hurt my current rankings. need your expert advice on this matter. thanks!

01-21-2012, 07:04 PM	#2
dburk Senior Warrior Member Join Date: Nov 2005 Location: Tampa, Florida Posts: 4,647 Thanks: 163 Thanked 673 Times in 583 Posts Contact Info	Re: robots.txt vs. htaccess Hi iamannie, Just use the Privacy feature in Wordpress Settings to prevent search engines from spidering your beta site until you are ready to launch.
	Don Burk * Get Results - Outsource Your PPC Management * Get a Keyword Domain Name - www.SeriousNames.com

01-21-2012, 07:05 PM	#3
Matt Ward Addicted to IM War Room Member Join Date: Oct 2010 Location: {Sunny\|Frigid} Canada Posts: 716 Thanks: 65 Thanked 150 Times in 89 Posts	Re: robots.txt vs. htaccess Robots.txt is only a suggestion; no bot HAS to follow it. You'd be much better off restricting it via htaccess if you really wanted to be secure.
	"Keep moving forward."

01-21-2012, 11:42 PM	#4
AlbertSmiths Active Warrior Join Date: Nov 2011 Location: uk Posts: 38 Thanks: 4 Thanked 1 Time in 1 Post	Re: robots.txt vs. htaccess Robots.txt - This tells the Search engines which URLs should not be indexed. .htaccess - This file can achieve many functions, such as - Ban a particular IP or User-agent. - Redirect URLs - Rewrite URLs with SE friendly names etc
	Ecommerce Solutions \| Ecommerce Web Design \| Web Design Northern Ireland

01-31-2012, 03:12 PM	#5
iamannie Warrior Member Join Date: Jan 2012 Posts: 3 Thanks: 0 Thanked 0 Times in 0 Posts	Re: robots.txt vs. htaccess thanks for the response guys!

01-31-2012, 05:15 PM	#7
paulgl Plundering the Web War Room Member Join Date: Feb 2007 Location: , , . Posts: 4,851 Thanks: 804 Thanked 1,200 Times in 887 Posts	Re: robots.txt vs. htaccess The answer is simpler than that. Just password protect the folder. Done and done. Very easy if you have cpanel. Robots.txt is iffy. Google is not the only place that indexes sites. In reality, there is no reason to NOT play around with it, live. Paul
	How to Make Money off Facebook: Login to your account. Deactivate your account. Get your butt to work.

02-02-2012, 02:37 AM	#8
smita Warrior Member Join Date: Feb 2012 Posts: 17 Thanks: 0 Thanked 0 Times in 0 Posts	Re: robots.txt vs. htaccess .htaccess is a configuration file which is used to restrict users from accessing your private pages. whereas Robots.txt is like a text file used by any websites owner to give to give instructions about their site to web robots.
	Cheap Hotels in Sligo

02-02-2012, 02:41 AM	#9
chintangurjar Warrior Member Join Date: Jan 2012 Posts: 14 Thanks: 0 Thanked 2 Times in 2 Posts	Re: robots.txt vs. htaccess Just use the WordPress Privacy Settings to prevent search engines from spidering the site beta until you are ready to launch.
	Affordable SEO Services

02-02-2012, 03:03 AM	#10
jonson Warrior Member Join Date: Jan 2012 Posts: 21 Thanks: 0 Thanked 2 Times in 2 Posts	Re: robots.txt vs. htaccess Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site. A .htaccess file is a directory-level configuration file supported by several web servers, that allows for decentralized management of web server configuration.The original purpose of .htaccess - reflected in its name was to allow per-directory access control, by for example requiring a password to access the content. Nowadays however, the .htaccess files can override many other configuration settings including content type and character set, CGI handlers, etc.
	water softeners