Wordpress SEO and crawling problem

13 replies
  • SEO
  • |
Hi,

Around 10 days ago I moved from the reliable Headspace 2 to Yoast's Wordpress SEO on the basis of rave reviews. Before moving, most of my pages were ranked well enough to give me a living from my site.

After the 'upgrade', traffic dropped off alarmingly and when I check Google using th 'site:' function, and click on 'cache' option I see that the site has not been crawled/indexed since Jan 29/30! Plus, my pages have dropped maybe 5 or 6 pages or more.

I assumed it was a sitemap failure and disabled the built-in XML and enabled the trusty Workpress plug in. Still, I am not being crawled.

Can anyone please help me. I am losing money hand over first.

Charlotte
#crawling #problem #seo #wordpress
  • Profile picture of the author Charlotteb
    PS. I have just installed Google Webmaster Tools, and see I that while I AM being crawled, I am also getting a LOT of crawl errors 404.

    Perplexed!
    {{ DiscussionBoard.errors[3333184].message }}
    • Profile picture of the author raviv
      Hi
      It may be that the new plugins may have altered the http response codes, not sure. If you are using Firefox, there is a plugin called livehttpheaders
      Then install this HEaderMonitor plugin. It will show you the header code on your browser status bar.

      Now check the list at Google for what the different codes stand for and make sure you dont get an unsavory header response. Try this and see what happens.

      Also did you change all the metatags when you changed the plugins?

      Best
      Raviv
      {{ DiscussionBoard.errors[3333242].message }}
    • Profile picture of the author pons_saravanan
      can you see the robots.txt file in your webiste. <your website>/robots.txt

      You can see this in the webmaster tools as well. all the bots should be allowed.
      I have seen few days before that, if you dont specify the robots.txt, the default robots txt is not allowing the bots.
      Note: you havent placed the robots.txt. but <your website>/robots.txt serves you a page, then you have to prepare a robots.txt and place in the wwwroot.
      {{ DiscussionBoard.errors[3333256].message }}
    • Profile picture of the author raviv
      Hi Charlotte
      If you do not have a robots.txt file, then it means by default the bots are allowed to spider your site. If you want to specifically block certain areas of your site, then you have to use the robots.txt file (create it first and upload it to your web server). Nothing to worry about if it is not there
      Best
      Raviv
      {{ DiscussionBoard.errors[3333336].message }}
  • Profile picture of the author hath80
    Try this and see what happens.
    {{ DiscussionBoard.errors[3333372].message }}
    • Profile picture of the author markowe
      Originally Posted by hath80 View Post

      Try this and see what happens.
      Another one! Try WHAT and see what happens?
      Signature

      Who says you can't earn money as an eBay affiliate any more? My stats say otherwise

      {{ DiscussionBoard.errors[3333383].message }}
      • Profile picture of the author Charlotteb
        Thanks for this. It appear I am eing crawled from WenmasterTools.

        Url Googlebot Googlebot-Mobile
        Allowed
        Detected as a directory; specific files may have different restrictions
        Allowed
        Detected as a directory; specific files may have different restrictions

        BUT. when I paste this in my browser:

        /w ww. etc - cant post links yet /robots.txt

        I get this:

        User-agent: *
        Disallow:Sitemap: sitemap.xml.gz[]

        Sorry this is so basic, I am not a real tecchie.

        C
        {{ DiscussionBoard.errors[3333552].message }}
        • Profile picture of the author markowe
          Originally Posted by Charlotteb View Post


          /w ww. etc - cant post links yet /robots.txt

          I get this:

          User-agent: *
          Disallow:Sitemap: sitemap.xml.gz[]

          Sorry this is so basic, I am not a real tecchie.

          C
          That's ALL that's in your robots.txt file?

          Hmm, that's two directives:

          User-agent: *
          Disallow:

          Sitemap: sitemap.xml.gz


          The first one basically means all robots are allowed access to all of your site (which you want). The second tells robots where your sitemap is (the square brackets aren't there in your file, right? That might cause a problem if they were) - that should be no problem either. Can you download the sitemap via:

          yoursite.com/sitemap.xml.gz ?

          Anyway, Google don't just knock sites down the SERPS because they couldn't find a sitemap all of a sudden, though MAYBE not being able to crawl would affect the rankings. Check the HTML header of your site and see if there are any directives like noindex in there for some strange reason, though AFAICS your pages ARE indexed still. Maybe you are just experiencing the Google Dance and it's coincidence? I am not sure just changing your SEO plugin should have such a drastic effect - unless the plugin has done something really stupid.

          Can't you either a) roll back to your old plugin or b) get on the plugin writer's back to tell you what's up?! (that's what my plugin users do!)
          Signature

          Who says you can't earn money as an eBay affiliate any more? My stats say otherwise

          {{ DiscussionBoard.errors[3333591].message }}
  • Profile picture of the author Charlotteb
    I have done the site+robots.text, and now get this:xxxxxx being my site name
    Does this mean my sitemap is being disallowed?

    User-agent: *
    Disallow:Sitemap: xxxxxxxxxxxxxxx/sitemap.xml.gz
    {{ DiscussionBoard.errors[3334071].message }}
    • Profile picture of the author markowe
      Originally Posted by Charlotteb View Post

      I have done the site+robots.text, and now get this:xxxxxx being my site name
      Does this mean my sitemap is being disallowed?

      User-agent: *
      Disallow:Sitemap: xxxxxxxxxxxxxxx/sitemap.xml.gz
      No, I don't believe so, the commands have just run onto one line. They should read as I quoted them earlier, with the 'Disallow:' on a separate line, but I doubt that makes a difference to the robot.

      However, just in case, you might want to separate out those declarations like I did above.

      Again, though I find it hard to believe this would be the cause. If Google likes a site they aren't going to knock you down the SERPS just because your sitemap is unavailable. Go hassle that plugin writer!
      Signature

      Who says you can't earn money as an eBay affiliate any more? My stats say otherwise

      {{ DiscussionBoard.errors[3334637].message }}
  • Profile picture of the author poWderKeg
    I haven't seen this brought up yet and hopefully it's not a dumb question, but did you change your URL structure when you installed the WordPress SEO plugin from Yoast?

    Have you also ported over your previous SEO Titles/Descriptions, etc.? (there are plugins to help do this)
    {{ DiscussionBoard.errors[3334669].message }}
    • Profile picture of the author Charlotteb
      Yes I transported all my settings using the Yoast import option within WP SEO.

      I have emailed the man himself and he is diagnosing it as i type.

      Fingers crossed

      C
      {{ DiscussionBoard.errors[3341097].message }}
      • Profile picture of the author Charlotteb
        Seems that Joost is unable to help me due to the site being of an adult nature. Can anyone else suggest anything?
        {{ DiscussionBoard.errors[3346651].message }}

Trending Topics