Robots.txt

[tyang]

So I've been using wordpress for much of my work lately and just noticed that the robots.txt is by default
User-agent: * Disallow:

Wouldn't this prevent search engines from indexing my site? Also I can't find where this file is?

Any help would be greatly appreciated.

[KenThompson]

I believe it's in your dashboard settings where you can find
the little check boxes that take care of that for you.

You can block all search engines, allow, etc.


Ken

[tyang]

Thanks Ken, I do have that option checked under the privacy settings. Just not sure why when I type robots.txt after my domain is says Disallow. Maybe not something to worry about?

[KenThompson]

Well that's interesting. I've dabbled with robots.txt files but not
too often.

Maybe Istvan will spot this and lend some words of wisdom. He's
pretty "up" on WP and blogs.

There is a robots.txt plugin you can get, set up and see what effect
that has. But, Istvan will yell at me for suggesting you get yet another
plugin. lol


Ken

[tyang]

Appreciate your help, from digging around some more, it looks like

User-agent: * Disallow:

actually allows search engine. If you want to disallow a search engine you would put it after the ":" Go figure

[KenThompson]

Originally Posted by tyang

Appreciate your help, from digging around some more, it looks like

User-agent: * Disallow:

actually allows search engine. If you want to disallow a search engine you would put it after the ":" Go figure

Oh well of course! LOL

I knew that, was just testing you.


Ken

[Istvan Horvath]

Originally Posted by tyang

just noticed that the robots.txt is by default
User-agent: * Disallow:

Originally Posted by tyang

Thanks Ken, I do have that option checked under the privacy settings.

So, then WHY are you surprized that it says: *Disallow???

If this is checked - I would like to block search engines, but allow normal visitors AND you add /robots.txt after your blogs URL - the disallow text will appear.

If you change the Privacy settings to - I would like my site to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers AND you add the /robots.txt after your URL... you will get a 404.

Provided your blog is set up normally...

Everything else is false

[Vanfenix]

Don't Trust the Robots. You've seen Terminator right?

no no - keyword your site is all you need.

[tyang]

Originally Posted by Istvan Horvath

So, then WHY are you surprized that it says: *Disallow???

If this is checked - I would like to block search engines, but allow normal visitors AND you add /robots.txt after your blogs URL - the disallow text will appear.

If you change the Privacy settings to - I would like my site to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers AND you add the /robots.txt after your URL... you will get a 404.

Provided your blog is set up normally...

Everything else is false

I was surprised by the disallow because there is a option for allow:, I didn't realize at the time that the disallow with the ":" is in fact allowing the search engines to spider my site. I Just have a site sitting for over three weeks without being index and I'm losing my mind checking everything. Thanks a lot for the reassurance, I've been a web designer for 4 years and never used robots.txt