Help With Robots.txt

Istvan Horvath

15 years ago

What is a robots.txt plugin?
What are you trying to do and why?

Thanks

Signature

{{ DiscussionBoard.errors[3182466].message }}

warriorspirit786

15 years ago

robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

I require this so google bots can index all my websites content.

Thanks
1 reply

{{ DiscussionBoard.errors[3183168].message }}

Lloyd Buchinski 15 years ago

Originally Posted by warriorspirit786

I require this so google bots can index all my websites content.

You don't actually require a robots file to help index content. The default of any bot is to follow. You only require a robots.text if you do not want them to follow or index.

A site map is what you would use to help a bot find all the pages. An example here: http://foothillsweb.ca/sitemap.xml

Edit: Oops, just looked at it and it hasn't been updated for a long long time.
- Thanks
Signature

Do something spectacular; be fulfilled. Then you can be your own hero. Prem Rawat

The KimW WSO
{{ DiscussionBoard.errors[3194143].message }}

Istvan Horvath

15 years ago

Seemingly, you have difficulties understanding what you read...

Don't quote me anything. I was asking what is a "robot.txt plugin". - Can you tell me? Since you installed already 2 "robots.txt plugins" you should be able to tell me what they are...

Also, I didn't ask you to quote the wikipedia (because I know what robots.txt files are and I know how to use them).
I wanted to see YOUR understanding of them and to figure out why are you messing with stuff that you have no idea about.

Thanks

Signature

{{ DiscussionBoard.errors[3183625].message }}

warriorspirit786

15 years ago

How about some advice then? instead of the criticism. You are the expert,so enlighten me???

[ 1 ] Thanks
1 reply

{{ DiscussionBoard.errors[3183792].message }}

Havenhood 15 years ago

Did that plug-in add a robots meta tag in your head section? View the source of your main page and check for a robots meta tag. I'm not a Wordpress King!
- Thanks
Signature

--= -Spazzle- =--
{{ DiscussionBoard.errors[3184250].message }}

warriorspirit786

15 years ago

Hi Havenhood,

Do you mean this?


<meta name="robots" content="index,follow,noodp,noydir"

Thanks

{{ DiscussionBoard.errors[3188938].message }}

Havenhood

15 years ago

Yes, that's what I meant. The reason I've asked to see that meta tag is to verify that it's not blocking engines from referencing your site. That looks fine. Now, point to your robots text (robots.txt) file and load it into your browser. It should be at the same location as your main page. You shouldn't have anything that's disallowing the engines.

User agents should look like this: User-agent: *

You may have some Disallows for sub directories, but it shouldn't look like this:
User-agent: *
Disallow: /

If you're not sure what it means, you may want to post the contents here. If that's okay, then how old is this site? Have you had engines come around before? Any additional information that you believe will help would be nice.

Thanks

Signature

--= -Spazzle- =--

{{ DiscussionBoard.errors[3189737].message }}

warriorspirit786

15 years ago

Hi,
I cannot find User-agent in my source file of website.

Thanks

{{ DiscussionBoard.errors[3192961].message }}

warriorspirit786

15 years ago

Ok, i am quite confused. I already have a sitemap for my site.

Can you please definately confirm that Robots.txt plugin is only required if i dont want google to find all my sites webpages? I was advised by someone i needed this for google bots.

Thanks

Thanks
1 reply

{{ DiscussionBoard.errors[3194189].message }}

csmcmanus

15 years ago

if you are wondering what you need for google to index all of your content of your site, why do you not just go over to google webmaster tools and get an account. once you submit your sitemap ( which is really all you need to do ) to the webmaster tools it will get crawled and if your site structure matches your sitemap then yes you will get all of your content indexed.

if google has problems with your site, your webmaster tools will be able to help you identify the problem and take care of it then.

Thanks
1 reply

{{ DiscussionBoard.errors[3195087].message }}

Lloyd Buchinski

15 years ago

Originally Posted by warriorspirit786

Can you please definately confirm that Robots.txt plugin is only required if i dont want google to find all my sites webpages? I was advised by someone i needed this for google bots.

Thanks

Years ago I saw a couple of sites that said you need do follow meta tags but at least from the authority sites now the info is that they are pointless. I did put my first few sites up with them, then took them all out later as a waste of bandwidth.

Here is an example of a meta no follow tag that I use on pages I want to limit to people I invite.

<meta name="robots" content="noindex,nofollow,noarchive" />

If you do want to include a do follow tag, just take out all the 'no's from it, and put it in the head. It won't do any harm besides the bandwidth and of course that isn't much.

But you probably just saw a bit of stray misinformation still floating around.

Originally Posted by csmcmanus

if you are wondering what you need for google to index all of your content of your site, why do you not just go over to google webmaster tools and get an account. once you submit your sitemap ( which is really all you need to do ) to the webmaster tools it will get crawled and if your site structure matches your sitemap then yes you will get all of your content indexed.

if google has problems with your site, your webmaster tools will be able to help you identify the problem and take care of it then.

That sounds good to me and fits in with everything I understand. No plug ins needed.

And btw, I don't think Istvan was being rude with his comments. Your first answer completely missed what he had asked and it might have led to a bit of exasperation. This communication stuff isn't easy. I consider that I'm still learning to talk.

That's nothing compared to some of my miss communications. I'm not the slightest bit religious but one prayer I can relate to is "Lord keep your arm around my shoulder, and your hand over my big mouth."