Robots.txt And Add-On Domain?

[MisterE]

I've never concerned myself in the past with a Robots.txt file being included with any of my domains, but recently I've been reviewing a couple of SEO ebooks that seem to imply that they are necessary for proper and efficient optimization.

After studying and reading up on robots text files I've managed to confuse myself thoroughly.

So, here's what I want to do:

I have a root domain 123.com

Addon domain is abc.com (abc.123.com)

I need to have a robots.txt file that will only allow permission to index the addon domain (abc.com), and not the main domain (123.com) it is used for personal files and server scripts. I do have a WordPress install on the root 123.com though.

I've created a new site in a subfolder of the addon domain abc.com/new site.

I want to give permission to the search engine bots to search this 'abc' 'root' folder and any other additional sites that I create in this directory abc.com/additional sites site.

Where would I place the 'robots.txt' file..... Under the root 123.com or under the addon site abc.com?

Again. I don't want the subfolders of the root domain with the exception of the WordPress install to be searched or indexed. Everything in all the addon domains will be fair game for the search engines.

And, could somebody give me an example of what the contents of a properly constructed robots.txt should be, other than 'Allow'?

Thanks,
Mark

[UKecommerce]

Would it not be more conflict-free to have abc.com/123 instead and simply block the 123 folder with your robots.txt file?

[MisterE]

My root domain '123.com' is what I registered with my shared hosting provider. I'm not sure if I could swap the two domains so that abc.com could become the primary root domain.

[UKecommerce]

My apologies, what I mean was rootdomain/subfolder, using the subfolder instead of a subdomain. It'd be much easier to administer with a robot.txt. However, perhaps another viewer may be able to help you further.

[MisterE]

Thanks UK for your response.

I believe that with shared hosting that all addon domains are actually subdomains of the root i.e. www.abc.123.com. I could be wrong about this but it is my understanding.

[socialbookmark]

In shared hosting, you have a primary domain and all other addon domains will be shown in your control panel and sub domains of it. and when you want to access them in file manager, addon domains will be shwon as .../addon1/...

[scott g]

Originally Posted by UKecommerce

My apologies, what I mean was rootdomain/subfolder, using the subfolder instead of a subdomain. It'd be much easier to administer with a robot.txt. However, perhaps another viewer may be able to help you further.

I guess I'm confused...

I have "mymain.com" with it's own robots.txt
I also have "subto.mymain.com" with it's own robots.txt
I also have "sutoto.mymain.com" with it's own robots.txt too...

Is this what you're talking about?!?

[MisterE]

Scott,

I'm most likely more confused than you.... I don't even have a single robots.txt at all.

What I keep running across is the statements that there can and should only be one robots.txt file per domain. With shared hosting plans the domain that you originally signed up with is the root domain.... On the servers side each addon domain appears as a subdomain of the root which is than redirected to the addon domains name in the simple format of www.addon .com with no reference to the original root domain.

What's throwing me is where to place this single robots text file, in the root domains folder? Or, in the addon domains folder?

If the file is placed in the root folder will it index everything, addon domains, subdomains, every folder and all of it's content?

[scott g]

Originally Posted by MisterE

Scott,

I'm most likely more confused than you.... I don't even have a single robots.txt at all.

What I keep running across is the statements that there can and should only be one robots.txt file per domain. With shared hosting plans the domain that you originally signed up with is the root domain.... On the servers side each addon domain appears as a subdomain of the root which is than redirected to the addon domains name in the simple format of www.addon .com with no reference to the original root domain.

What's throwing me is where to place this single robots text file, in the root domains folder? Or, in the addon domains folder?

If the file is placed in the root folder will it index everything, addon domains, subdomains, every folder and all of it's content?

I don't know if this answers your question, but I'll type it anyway!

I have a Baby Hosting Plan w/ hostgator. The main main domain is "something.net" that I don't even use anymore (except for some backlinks) and it has a robots.txt file too. This is what I'm doing for my site that I'm using subdomains with:

keyword.org/robots.txt
"Sitemap: http://keyword.org/sitemap.xml
User-agent: *
Disallow: /cgi-bin/"
sub.keyword.org/robots.txt
"Sitemap: http://sub.keyword.org/sitemap.xml
User-agent: *
Disallow: /cgi-bin/"sub2.keyword.org/robots.txt
"Sitemap: http://sub2.keyword.org/sitemap.xml
User-agent: *
Disallow: /cgi-bin/"

If this helps you at all then good, if not maybe it will help someone else!

CHEERS & HAPPY THANKSGIVING!

[MisterE]

Scott,

The way you are implementing the robots txt seems logical to me.

But I'm still being thrown off by the statement that I've run into of there being only one instance of the file for your, our, shared hosting plans.

I'll most likely set everything the way you are and change it later if I find out differently.

thanks,
Mark

[scott g]

Originally Posted by MisterE

Scott,

The way you are implementing the robots txt seems logical to me.

But I'm still being thrown off by the statement that I've run into of there being only one instance of the file for your, our, shared hosting plans.

I'll most likely set everything the way you are and change it later if I find out differently.

thanks,
Mark

A subdomain is treated just like any other domain - it is itself. On my MAIN.COM I have Wordpress installed - on the SUBS.MAIN.COM I have static HTML/CSS sites.

In my Google Analytics and Webmaster Tools each domain and subdomain is reported/tracked individually ("+ new site").

In my Webmaster Tools, I can see that Google crawled and found and reported on the robots.txt of each domain and subdomain site I have added to my Webmaster Tools. It shows you the content of each Robots.txt too so I know that EACH domain/subdomain's ROBOTS.TXT is crawled being crawled and registered. The Crawler access tab which can be found under Site Configuration in Google Webmaster Tools.

If your site has content you don't want Google or other search engines to access, use a robots.txt file to specify how search engines should crawl your site's content.
If private or out-of-date content is appearing in Google search results, use the Removal URL tool to request its removal (but first, make sure to review our removal requirements).

[scott g]

Originally Posted by MisterE

Scott,

The way you are implementing the robots txt seems logical to me.

But I'm still being thrown off by the statement that I've run into of there being only one instance of the file for your, our, shared hosting plans.

I'll most likely set everything the way you are and change it later if I find out differently.

thanks,
Mark

Please, disregard what you think you know. It will make things easier!

[MisterE]

Thanks Scott!

I will do as you suggest and not worry myself any further.

It does make sense to me the way you have implemented robots.txt.

Mark