Wordpress SEO and crawling problem

[Charlotteb]

Hi,

Around 10 days ago I moved from the reliable Headspace 2 to Yoast's Wordpress SEO on the basis of rave reviews. Before moving, most of my pages were ranked well enough to give me a living from my site.

After the 'upgrade', traffic dropped off alarmingly and when I check Google using th 'site:' function, and click on 'cache' option I see that the site has not been crawled/indexed since Jan 29/30! Plus, my pages have dropped maybe 5 or 6 pages or more.

I assumed it was a sitemap failure and disabled the built-in XML and enabled the trusty Workpress plug in. Still, I am not being crawled.

Can anyone please help me. I am losing money hand over first.

Charlotte

[Charlotteb]

PS. I have just installed Google Webmaster Tools, and see I that while I AM being crawled, I am also getting a LOT of crawl errors 404.

Perplexed!

[raviv]

Hi
It may be that the new plugins may have altered the http response codes, not sure. If you are using Firefox, there is a plugin called livehttpheaders
Then install this HEaderMonitor plugin. It will show you the header code on your browser status bar.

Now check the list at Google for what the different codes stand for and make sure you dont get an unsavory header response. Try this and see what happens.

Also did you change all the metatags when you changed the plugins?

Best
Raviv

[pons_saravanan]

can you see the robots.txt file in your webiste. <your website>/robots.txt

You can see this in the webmaster tools as well. all the bots should be allowed.
I have seen few days before that, if you dont specify the robots.txt, the default robots txt is not allowing the bots.
Note: you havent placed the robots.txt. but <your website>/robots.txt serves you a page, then you have to prepare a robots.txt and place in the wwwroot.

[raviv]

Hi Charlotte
If you do not have a robots.txt file, then it means by default the bots are allowed to spider your site. If you want to specifically block certain areas of your site, then you have to use the robots.txt file (create it first and upload it to your web server). Nothing to worry about if it is not there
Best
Raviv

[hath80]

Try this and see what happens.

[markowe]

Originally Posted by hath80

Try this and see what happens.

Another one! Try WHAT and see what happens?

[Charlotteb]

Thanks for this. It appear I am eing crawled from WenmasterTools.

Url Googlebot Googlebot-Mobile
Allowed
Detected as a directory; specific files may have different restrictions
Allowed
Detected as a directory; specific files may have different restrictions

BUT. when I paste this in my browser:

/w ww. etc - cant post links yet /robots.txt

I get this:

User-agent: *
Disallow:Sitemap: sitemap.xml.gz[]

Sorry this is so basic, I am not a real tecchie.

C

[markowe]

Originally Posted by Charlotteb

/w ww. etc - cant post links yet /robots.txt

I get this:

User-agent: *
Disallow:Sitemap: sitemap.xml.gz[]

Sorry this is so basic, I am not a real tecchie.

C

That's ALL that's in your robots.txt file?

Hmm, that's two directives:

User-agent: *
Disallow:

Sitemap: sitemap.xml.gz

The first one basically means all robots are allowed access to all of your site (which you want). The second tells robots where your sitemap is (the square brackets aren't there in your file, right? That might cause a problem if they were) - that should be no problem either. Can you download the sitemap via:

yoursite.com/sitemap.xml.gz ?

Anyway, Google don't just knock sites down the SERPS because they couldn't find a sitemap all of a sudden, though MAYBE not being able to crawl would affect the rankings. Check the HTML header of your site and see if there are any directives like noindex in there for some strange reason, though AFAICS your pages ARE indexed still. Maybe you are just experiencing the Google Dance and it's coincidence? I am not sure just changing your SEO plugin should have such a drastic effect - unless the plugin has done something really stupid.

Can't you either a) roll back to your old plugin or b) get on the plugin writer's back to tell you what's up?! (that's what my plugin users do!)

[Charlotteb]

I have done the site+robots.text, and now get this:xxxxxx being my site name
Does this mean my sitemap is being disallowed?

User-agent: *
Disallow:Sitemap: xxxxxxxxxxxxxxx/sitemap.xml.gz

[markowe]

Originally Posted by Charlotteb

I have done the site+robots.text, and now get this:xxxxxx being my site name
Does this mean my sitemap is being disallowed?

User-agent: *
Disallow:Sitemap: xxxxxxxxxxxxxxx/sitemap.xml.gz

No, I don't believe so, the commands have just run onto one line. They should read as I quoted them earlier, with the 'Disallow:' on a separate line, but I doubt that makes a difference to the robot.

However, just in case, you might want to separate out those declarations like I did above.

Again, though I find it hard to believe this would be the cause. If Google likes a site they aren't going to knock you down the SERPS just because your sitemap is unavailable. Go hassle that plugin writer!

[poWderKeg]

I haven't seen this brought up yet and hopefully it's not a dumb question, but did you change your URL structure when you installed the WordPress SEO plugin from Yoast?

Have you also ported over your previous SEO Titles/Descriptions, etc.? (there are plugins to help do this)

[Charlotteb]

Yes I transported all my settings using the Yoast import option within WP SEO.

I have emailed the man himself and he is diagnosing it as i type.

Fingers crossed

C

[Charlotteb]

Seems that Joost is unable to help me due to the site being of an adult nature. Can anyone else suggest anything?