9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

over 3000 crawl errors overnight.

by AdultMySpace
7 replies
Hoping someone could provide some advice here.

My site was starting to rank quite nicely for some competitive search terms and I was seeing quite a bit of traffic. Now the initial website build had a forum I was going to use, although it never really went live or was linked to on the site.

It was left on the backend by the developer and the forum, as expected without moderation, has been spammed to hell and back. Xrumer strikes again! I only came across it by chance myself.

Now in my haste, I asked the developer to remove the forum which has now resulted in over 3000 crawl errors on my site! Needless to say, I am not even on the first 500 google search results and its happened almost overnight. My traffic has dropped to almost zero.

Deleting the forums was leading to the 404 page, which I have redirected to the landing page for now hoping that this is going to solve the problem. Google still has the crawl errors and I am hoping the next bot visit will see the end of that.

What would be the best way for me to rectify this problem? The developer I have used while great at design is a bit lacking when it comes to SEO in particular. Im a bit confused if I should just leave it as it was, redirect, create a better redirect page, use a 301????

Thanks in advance
#crawl #errors #overnight
  • Profile picture of the author KenThompson
    Just out of curiosity, I would put the forum back and clean it up
    just to see what happens. What I'm not quite getting is you said
    there were no links between the forum and your main site. Yes?

    If that's the case, it would seem that removing the forum, or taking
    it down, should not have caused the massive drop in your ranking.

    Perhaps your site dropped for another reason. Interesting problem,
    though.

    Ken
    {{ DiscussionBoard.errors[2418169].message }}
  • Profile picture of the author Dennis Gaskill
    Use 301 redirects for the broken links and build a new XML Sitemap. Then give it a couple months to get corrected.

    If that doesn't do it after a couple months, log into your webmaster account and apply for reconsideration. That's usually reserved for sites that get penalized, but I used it when my site was hacked and the hacker's faulty script (he was trying to siphon off my traffic) resulted in Google thinking there were thousands of broken links on my site.

    Tell your story to them, and wait. It will probably take 3 or 4 more months at this point, and they won't let you know if they've done anything or not, but if your site is clean your rankings should pick back up.

    Hopefully you won't have to go to the second step, the 301's and Sitemap will do the trick.
    Signature

    Just when you think you've got it all figured out, someone changes the rules.

    {{ DiscussionBoard.errors[2418172].message }}
    • Profile picture of the author KenThompson
      Originally Posted by Dennis Gaskill View Post

      Use 301 redirects for the broken links and build a new XML Sitemap. Then give it a couple months to get corrected.

      If that doesn't do it after a couple months, log into your webmaster account and apply for reconsideration. That's usually reserved for sites that get penalized, but I used it when my site was hacked and the hacker's faulty script (he was trying to siphon off my traffic) resulted in Google thinking there were thousands of broken links on my site.

      Tell your story to them, and wait. It will probably take 3 or 4 more months at this point, and they won't let you know if they've done anything or not, but if your site is clean your rankings should pick back up.

      Hopefully you won't have to go to the second step, the 301's and Sitemap will do the trick.
      Wow. I'd wanna shoot someone. lol (not really)

      What a shame. Sorry it happened, man.

      Dennis... I knew the rankings would take time, but I had no idea it
      would take that long, or cause a problem to that degree.

      Ken
      {{ DiscussionBoard.errors[2418177].message }}
    • Profile picture of the author AdultMySpace
      Originally Posted by Dennis Gaskill View Post

      Use 301 redirects for the broken links and build a new XML Sitemap. Then give it a couple months to get corrected.

      If that doesn't do it after a couple months, log into your webmaster account and apply for reconsideration. That's usually reserved for sites that get penalized, but I used it when my site was hacked and the hacker's faulty script (he was trying to siphon off my traffic) resulted in Google thinking there were thousands of broken links on my site.

      Tell your story to them, and wait. It will probably take 3 or 4 more months at this point, and they won't let you know if they've done anything or not, but if your site is clean your rankings should pick back up.

      Hopefully you won't have to go to the second step, the 301's and Sitemap will do the trick.
      My biggest frustration here is that I have outsourced so much of the backlinking and it was seeing steady results. 3-4 months is way too long to get back to where I was as I am running a subscription based membership site!

      Im at the stage where I am wondering if its just not worthwhile registering a new domain and starting from scratch, although I lose all those backlinks and basic branding, not to mention a very good domain thats perfect for the words I am targetting.

      Im very frustrated at the moment. It was a simple mistake the developer made and these things happen but the bottom line here is that it has hammered my site
      Signature

      {{ DiscussionBoard.errors[2418189].message }}
  • Profile picture of the author AdultMySpace
    The problem was that the forum was live and hosted on the domain, it just had no physical link from the pages which were live. The only way it could have been found is if someone used one of the scraping tools and found it. Which is clearly what happened, because when I cam across it, it was spammed to hell and back with Russian viagra and all sorts of rubbish.

    Now the developer made a mistake with the site and these things happen. Putting thr forum back up wont help because he deleted everything and google is pointing to member ID's and such. They have basically spammed other forums to index their spam profiles and posts.

    I guess I need to know how I can go about ensuring that I am not penalised for removing content from my site, even though that content was indexed. I was front page and top 5 for some terms and the minute that forum came down I have fallen off the radar completely.

    I am also wondering if maybe the reason why I did not rank so highly was because the inbound linking for the site was through the roof with the forum. I am very tempted to try this again with a test site and just leave it up, seeing it does anything for ranking with minimal search terms
    Signature

    {{ DiscussionBoard.errors[2418179].message }}
    • Profile picture of the author seasoned
      Originally Posted by AdultMySpace View Post

      The problem was that the forum was live and hosted on the domain, it just had no physical link from the pages which were live. The only way it could have been found is if someone used one of the scraping tools and found it. Which is clearly what happened, because when I cam across it, it was spammed to hell and back with Russian viagra and all sorts of rubbish.

      Now the developer made a mistake with the site and these things happen. Putting thr forum back up wont help because he deleted everything and google is pointing to member ID's and such. They have basically spammed other forums to index their spam profiles and posts.

      I guess I need to know how I can go about ensuring that I am not penalised for removing content from my site, even though that content was indexed. I was front page and top 5 for some terms and the minute that forum came down I have fallen off the radar completely.

      I am also wondering if maybe the reason why I did not rank so highly was because the inbound linking for the site was through the roof with the forum. I am very tempted to try this again with a test site and just leave it up, seeing it does anything for ranking with minimal search terms
      It's impossible for a scrapping tool to find the forum unless there is a link on an earlier page, or it is referenced elsewhere, and then GOOGLE could find it. The backend to google IS a scraper after all. It usually isn't mentioned, but a routine uses that, parses pages, and then uses it for THOSE pages and THAT routine is called a SPIDER. It is called that because it is almost like a spider traversing the net which is, itself, almost like a spider WEB. ALSO, some sites show pages visited, and your pages could then be indexed even if YOU didn't plan it.

      Steve
      {{ DiscussionBoard.errors[2419466].message }}
  • Profile picture of the author AdultMySpace
    Quick update on this

    Used proper 301 redirects and the end result of that was over 250k links to my site. The Xrumer spammers obviously built links to their links, which ended up becoming links built to my redirect. Not relevant links but I dont think they hurt my PR, thats for sure. The crawl errors gradually dropped away.

    I bounced back and have held number 1 for the last 3 months on a term with over 1.2 million pages competing. Quite interesting and I am very tempted to try this again with another site.

    1 - Put up a forum
    2 - Let it get spammed to death
    3 - Close it down after redirects in place
    Signature

    {{ DiscussionBoard.errors[3084082].message }}

Trending Topics