Submitting large sitemaps to Google

by 6 replies
We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.
#search engine optimization #google #large #sitemaps #submitting
Avatar of Unregistered
  • Profile picture of the author yukon
    Originally Posted by drneville View Post

    We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.



    Don't add 15 million URLs to a single xml file.

    You need to use a sitemap index xml file that includes multiple xml sitemaps, example, each category could have their own xml sitemap.

    Add a master.xml file to your robots.txt file.
    • Sitemap: hxxp://domain.com/sitemap/master.xml

    The master.xml file will contain URLs to each of your category.xml files (Google link example above).
    Signature
    We have the technology.
    • Profile picture of the author drneville
      Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.
  • Profile picture of the author waterguy
    It's been a while but there was a stat a few years ago if there are more that 10% broken links in your sitemap the bots will not follow them, so make sure your links all resolve to a 200 status, eliminate redirects and duplicate content to save crawl budget, and submit just the sitemap-index file to Google and Bing.

    You can check the link status by uploading or entering the sitemap index file URL in Screaming Frog.
  • Profile picture of the author Amanjit
    Banned
    [DELETED]
  • Profile picture of the author Anna Afrina
    [DELETED]

Trending Topics