8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

Submitting large sitemaps to Google

by drneville
6 replies
  • SEO
    • |
We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.
#google #large #sitemaps #submitting
  • Profile picture of the author yukon
    Banned
    Originally Posted by drneville View Post

    We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.



    Don't add 15 million URLs to a single xml file.

    You need to use a sitemap index xml file that includes multiple xml sitemaps, example, each category could have their own xml sitemap.

    Add a master.xml file to your robots.txt file.
    • Sitemap: hxxp://domain.com/sitemap/master.xml

    The master.xml file will contain URLs to each of your category.xml files (Google link example above).
    {{ DiscussionBoard.errors[11074027].message }}
    • Profile picture of the author drneville
      Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.
      {{ DiscussionBoard.errors[11074051].message }}
      • Profile picture of the author yukon
        Banned
        Originally Posted by drneville View Post

        Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.

        That's ok, Google won't index/crawl all the URLs at one time anyways.
        {{ DiscussionBoard.errors[11074067].message }}
  • Profile picture of the author waterguy
    It's been a while but there was a stat a few years ago if there are more that 10% broken links in your sitemap the bots will not follow them, so make sure your links all resolve to a 200 status, eliminate redirects and duplicate content to save crawl budget, and submit just the sitemap-index file to Google and Bing.

    You can check the link status by uploading or entering the sitemap index file URL in Screaming Frog.
    {{ DiscussionBoard.errors[11074104].message }}
    • Profile picture of the author drneville
      But as far as you know, there is no reason to not submit all of the URLs at once?
      {{ DiscussionBoard.errors[11074134].message }}
      • Profile picture of the author waterguy
        If the pages are unique and valuable then no problem, if they are thin content pages then don't bother. But the way you submit them all is submit the index file URL via Google Search Console. Google will crawl it and add the other individual xml sitemap files. Then it could weeks for Google to crawl all the HTML pages.
        {{ DiscussionBoard.errors[11074223].message }}

Trending Topics