Robots.txt

by tyang

Posted: 15 years ago 8 replies

So I've been using wordpress for much of my work lately and just noticed that the robots.txt is by default
User-agent: * Disallow:

Wouldn't this prevent search engines from indexing my site? Also I can't find where this file is?

Any help would be greatly appreciated.

#robotstxt

KenThompson 15 years ago

I believe it's in your dashboard settings where you can find
the little check boxes that take care of that for you.

You can block all search engines, allow, etc.

Ken
- Thanks
{{ DiscussionBoard.errors[3803954].message }}

tyang

15 years ago

Thanks Ken, I do have that option checked under the privacy settings. Just not sure why when I type robots.txt after my domain is says Disallow. Maybe not something to worry about?

Thanks
1 reply

{{ DiscussionBoard.errors[3803972].message }}

Istvan Horvath

15 years ago

Originally Posted by tyang

just noticed that the robots.txt is by default
User-agent: * Disallow:

Originally Posted by tyang

Thanks Ken, I do have that option checked under the privacy settings.

So, then WHY are you surprized that it says: *Disallow???

If this is checked - I would like to block search engines, but allow normal visitors AND you add /robots.txt after your blogs URL - the disallow text will appear.

If you change the Privacy settings to - I would like my site to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers AND you add the /robots.txt after your URL... you will get a 404.

Provided your blog is set up normally...

Everything else is false

Thanks
2 replies

Signature

{{ DiscussionBoard.errors[3804039].message }}

Vanfenix 15 years ago

Don't Trust the Robots. You've seen Terminator right?

no no - keyword your site is all you need.
- Thanks
Signature

If you need a website, something cool, slick, and affordable - something with built in aweber, hosting included, extremely easy to use. Unlimited pages, Drag and drop functionality. E-mail me. vanfenix1 at gmail.com. || I'll set you up a site for 15 days to test out || Squeeze pages? no problem. Lightweight E-commerce? easy as pie. My Websites are not like you've ever seen. Try it today FREE!
{{ DiscussionBoard.errors[3804063].message }}

tyang

15 years ago

Originally Posted by Istvan Horvath

I was surprised by the disallow because there is a option for allow:, I didn't realize at the time that the disallow with the ":" is in fact allowing the search engines to spider my site. I Just have a site sitting for over three weeks without being index and I'm losing my mind checking everything. Thanks a lot for the reassurance, I've been a web designer for 4 years and never used robots.txt

Thanks

{{ DiscussionBoard.errors[3804134].message }}

KenThompson 15 years ago

Well that's interesting. I've dabbled with robots.txt files but not
too often.

Maybe Istvan will spot this and lend some words of wisdom. He's
pretty "up" on WP and blogs.

There is a robots.txt plugin you can get, set up and see what effect
that has. But, Istvan will yell at me for suggesting you get yet another
plugin. lol

Ken
- [ 1 ] Thanks
{{ DiscussionBoard.errors[3803987].message }}

tyang

15 years ago

Appreciate your help, from digging around some more, it looks like

User-agent: * Disallow:

actually allows search engine. If you want to disallow a search engine you would put it after the ":" Go figure