SEO and indexing and robots.txt
- SEO |
Hi Everybody,
As usual, every SEO article and post I read has answered 1 question or taught me 1 thing and begged significantly more questions. Such is learning I guess. Here are my new questions.
I had never heard about robots.txt, but it seemed important, so I began looking it up. My google searching made it look so simple, until I began looking at other people's robots.txt. They specify bots and do all kinds of things. My site is very simple and I won't have much I don't want crawled. Should I just have a folder that I don't want indexed and do a generic useragents:* Disallow: /noindex/ or is there a better way to do that? How about an images folder, does it matter if that is indexed?
Also, how about pages with little to no content, like a contact page with a few sentences and a form? Would it be better not to index that? I really want to develop a content based site with organic growth, so I don't want low content pages to negatively affect my ranking. If I should avoid indexing such pages, should it be with a noindex meta tag or in the robots.txt or both?
Thank you for your help. I seem to find more to learn every time I feel like I am starting to get a grip on all of this.
David
As usual, every SEO article and post I read has answered 1 question or taught me 1 thing and begged significantly more questions. Such is learning I guess. Here are my new questions.
I had never heard about robots.txt, but it seemed important, so I began looking it up. My google searching made it look so simple, until I began looking at other people's robots.txt. They specify bots and do all kinds of things. My site is very simple and I won't have much I don't want crawled. Should I just have a folder that I don't want indexed and do a generic useragents:* Disallow: /noindex/ or is there a better way to do that? How about an images folder, does it matter if that is indexed?
Also, how about pages with little to no content, like a contact page with a few sentences and a form? Would it be better not to index that? I really want to develop a content based site with organic growth, so I don't want low content pages to negatively affect my ranking. If I should avoid indexing such pages, should it be with a noindex meta tag or in the robots.txt or both?
Thank you for your help. I seem to find more to learn every time I feel like I am starting to get a grip on all of this.
David
First Draft's: Download | Add-Ons | Templates | Purchase | Support | Affiliates