3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

Best way to get Google crawl a URL previously 410

by sayers
3 replies
  • SEO
    • |
Hi. Clue is in the title really.

I recently had a major bug on my classifieds site which caused millions of URLs to be generated. These URLs were 'incorrect', often assigning the wrong category directory with an unrelated subcategory for example. Similarly, the site works using faceted navigation, and due to the vast amount of pages on the site I set a limit for googlebot to only be able to crawl a max of 3 at any one time, anything past this should not be crawlable.

Unfortunately there were some complications in the development stage and number of 'correct' URLs were assigned a 410 header response. This issue was fixed last week, via a combination of 200 and 310, but I am not seeing those 'Not Found' errors dropping in WMT. My worry is that Google has visited the URLs, seen the 410, so will not visit those URLs again. Obviously this is not good!

Due to the high volume it is not feasible to use the Fetch as Google function for all.

Any smart ideas or advice greatly appreciated. Thanks.
#410 #crawl #google #previously #url
  • Profile picture of the author alexjames212
    I think you should perform a site index check to sort out the urls that are currently indexed by google. You can also try creating a new sitemap and submitting it in GWT. Also see that you disallow other malicious URLs using robots,txt.
    Signature
    French Bulldog Care
    {{ DiscussionBoard.errors[9549907].message }}
  • Profile picture of the author mariajames1
    I think you can create a new Sitemap for Google to index all of your URL's
    {{ DiscussionBoard.errors[9550340].message }}
  • Profile picture of the author kishoreseo
    Now generate new xml sitemap and then submit this new xml sitemap in WMT. once added in WMT then start fetching your urls in WMT to get index, few may get index fastly but few may take time
    {{ DiscussionBoard.errors[9550345].message }}

Trending Topics