Stupid, Stupid, Stupid! I Just Found Out Why My Sites Weren't Being Indexed by Google

[sashas]

I had set the privacy settings in Wordpress to "Open to visitors, but closed to search engines".

There. That little thing probably cost me $1k/month.

By default, all my wordpress blogs used to be open to everyone, search engines included. Now they're all closed to search engines by default, which, of course, makes indexing them rather impossible. Some plugin must've triggered this change in the default settings.

(If you don't understand what I'm talking about, go to 'Settings -> Privacy' in your wordpress admin panel. You'll see that there are two options. On my blogs, option 2 (closed to search engines) had been selected by default).

So while I was tearing my hair out trying to figure out why my sites have got just one page indexed in Google after more than a month and hundreds of backlinks, I overlooked this single tiny aspect that made my sites inaccessible to all spiders.

I suggest that you guys check out your Wordpress blog's privacy settings too to see if they aren't blocking spiders.

[Jason Johns]

Hi there,

I don't believe it's a plugin that has made this change. I think it's a change in Wordpress to the default settings! It's frustrating and I think many of us have been caught out by this.

Thanks for reminding us all about it!

All the best

Jason

[mmixon]

Learned something new and important today, (and it is still early) Thanks Sashas!

[Jacer]

I have installed both the most current release and the previous release of wordpress recently. I am sure "open to all" was the default setting in both cases. Maybe you accidently changed it?

[Solidsnake]

I checked mine (just installed last week) and the first one is ticked.. no problem at all..

new blogs are hard to getting indexed..

[Fraggler]

Default is definetly now 2nd option (clock search engines).

[turbohips]

mine was fine and didn't have block search engines by default. The first option was selected

[andybeveridge]

This has happend to me only a couple of days ago. Its set that way as default.
I am pretty sure it didn't cost me $$$$ though.

[Marhelper]

Yeah, I made the same dumb mistake myself a while ago. Good thing you caught it now.

[sashas]

I just did a check on all my blogs.

Blogs that were made before the middle of September have the first option enabled (open to search engines). Blogs made after that had the second option selected by default.

Heads up to everyone. All your marketing efforts will be of no use if the second option is selected.

[mfleisch]

Thanks sashas... been using mostly blogger and recently switched to WordPress. I didn't realize what the impact of not turning on this option would be!!!

[WareTime]

I think they do it by default so you have time to mess around with permalinks and don't have the default posts indexed before you get around to looking at those things.

On an aside a friend of mine that runs WP has had great luck with no permalinks just the default it ships with. I'd like to think one day we'll see no more /really-long-and-boring-spammy-way-too-wordy-thing-here/ urls.

[Mike Johnson]

Yep, I can confirm that a blog I installed several days ago using fantastico was blocking search engines by default.

I checked all of my other blogs (which I've recently updated to the current version) and they weren't blocking anything.

[nettech]

Its quite an easy mistake to make. I used to put this setting on when I was testing my sites pre-launch. i did forget to put activate it once or twice but you do rmeember after a while. Thanks for the reminder...lol

[Crafty Blogger]

Yeah, I've noticed this problem recently too. It's really annoying, especially if the bots get to the blog before I remember to change the default settings on a new blog to allowing them to visit - it keeps the site from being indexed for an even longer period of time!

[lanta99]

Luckily I caught this thread before I launched my wordpress network. Anyways, do you know if this is the same setting on wordpress MU? If so, I'll be sure to watch out for that.

[sashas]

Originally Posted by lanta99

Luckily I caught this thread before I launched my wordpress network. Anyways, do you know if this is the same setting on wordpress MU? If so, I'll be sure to watch out for that.

Haven't used WPMU so far, but there's no harm in checking.

[jasonmorgan]

raises hand... guilty as well.

I've done that before in the past as well as forget about building a sitemap, changing wordpress permalinks and some other silly things.

[AJsVRE]

Originally Posted by sashas

Some plugin must've triggered this change in the default settings.

There where a lot of people that had found out about this, and made posts on marketing forums and sent emails to their lists about this change. Sorry you hadn't seen it from one of those sources

It is NOT a plugin, it is now the DEFAULT SETTING if you install wordpress through fantastico. If you install wordpress manually (ftp'ing it to your server, manually setting up a database and going to the install config page - forgive me but I forget exactly what it's called, but if you've ever installed WP manually you know what I mean) then you don't have to worry, because when you do so, on the config page the option for that setting is right there.

[Sleaklight]

Thanks for posting this tread. Google had been indexing my website but it was only indexing the URL's and not the descriptions. I read your post and went to check my settings and was in the same boat as you! I too feel stupid! Thanks for making this post, I appreciate it.

[bigcat1967]

Thanks for posting this. Wondering if this applies to other Web 2.0 properties.