Submitting large sitemaps to Google

by drneville 6 replies
We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.
#search engine optimization #google #large #sitemaps #submitting
Avatar of Unregistered
  • Profile picture of the author yukon
    Originally Posted by drneville View Post

    We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.



    Don't add 15 million URLs to a single xml file.

    You need to use a sitemap index xml file that includes multiple xml sitemaps, example, each category could have their own xml sitemap.

    Add a master.xml file to your robots.txt file.
    • Sitemap: hxxp://domain.com/sitemap/master.xml

    The master.xml file will contain URLs to each of your category.xml files (Google link example above).
    Signature
    Hi
    {{ DiscussionBoard.errors[11074027].message }}
    • Profile picture of the author drneville
      Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.
      {{ DiscussionBoard.errors[11074051].message }}
      • Profile picture of the author yukon
        Originally Posted by drneville View Post

        Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.

        That's ok, Google won't index/crawl all the URLs at one time anyways.
        Signature
        Hi
        {{ DiscussionBoard.errors[11074067].message }}
  • Profile picture of the author waterguy
    It's been a while but there was a stat a few years ago if there are more that 10% broken links in your sitemap the bots will not follow them, so make sure your links all resolve to a 200 status, eliminate redirects and duplicate content to save crawl budget, and submit just the sitemap-index file to Google and Bing.

    You can check the link status by uploading or entering the sitemap index file URL in Screaming Frog.
    {{ DiscussionBoard.errors[11074104].message }}
    • Profile picture of the author drneville
      But as far as you know, there is no reason to not submit all of the URLs at once?
      {{ DiscussionBoard.errors[11074134].message }}
      • Profile picture of the author waterguy
        If the pages are unique and valuable then no problem, if they are thin content pages then don't bother. But the way you submit them all is submit the index file URL via Google Search Console. Google will crawl it and add the other individual xml sitemap files. Then it could weeks for Google to crawl all the HTML pages.
        {{ DiscussionBoard.errors[11074223].message }}
Avatar of Unregistered

Trending Topics

  • 6 {{ upvoteCount | shortNum }}

    Don't ever give up trying to make money online

    Isaack101 in Internet Marketing

    Hi there fellow Marketer... I WILL NOT SELL YOU ANYTHING HERE! I have been an online Marketer for 10 years now and I do this full-time. The first four years, ... [read more]

  • 4 {{ upvoteCount | shortNum }}

    What You See? When Choosing an Expired Domain for Making PBN.

    Mominul Islam in SEO

    What You See? When Choosing an Expired Domain for Making PBN. Need expert suggestion. Thanks

  • 1 {{ upvoteCount | shortNum }}

    Email Marketing <help me please with this CMM-X-Message-Delivery: Vj0xLjE7RD0yO0dEPTI7U0NMPTQ=>

    rosaMond in Email Marketing

    this days i receive some spam message on hotmail wihch have a CMM-X-Message-Delivery: Vj0xLjE7RD0yO0dEPTI7U0NMPTQ= My question is how they make this CMM-X-Message-Delivery cuz i'm an new email marketer and i ... [read more]

  • 4 {{ upvoteCount | shortNum }}

    Review my first Wordpress Website

    powermatrix in Web Design

    Hello Brothers, I have been a freelancer from 2010 to 2012 doing logo designing and slogan creations and ran a blogger site which i later deleted. Since then i have ... [read more]

  • 1 {{ upvoteCount | shortNum }}

    Which is better between Easy Digital Download (EDD) Vs. Product dyno for plugin protection

    I have these few questions to ask about these two products concerning plugin licensing. 1. Which of them is better for protecting plugin and licensing? 2. Does Easy Digital Download ... [read more]