Submitting large sitemaps to Google

by drneville 6 replies
We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.
#search engine optimization #google #large #sitemaps #submitting
Avatar of Unregistered
  • Profile picture of the author yukon
    Originally Posted by drneville View Post

    We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.



    Don't add 15 million URLs to a single xml file.

    You need to use a sitemap index xml file that includes multiple xml sitemaps, example, each category could have their own xml sitemap.

    Add a master.xml file to your robots.txt file.
    • Sitemap: hxxp://domain.com/sitemap/master.xml

    The master.xml file will contain URLs to each of your category.xml files (Google link example above).
    Signature
    Hi
    {{ DiscussionBoard.errors[11074027].message }}
    • Profile picture of the author drneville
      Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.
      {{ DiscussionBoard.errors[11074051].message }}
      • Profile picture of the author yukon
        Originally Posted by drneville View Post

        Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.

        That's ok, Google won't index/crawl all the URLs at one time anyways.
        Signature
        Hi
        {{ DiscussionBoard.errors[11074067].message }}
  • Profile picture of the author waterguy
    It's been a while but there was a stat a few years ago if there are more that 10% broken links in your sitemap the bots will not follow them, so make sure your links all resolve to a 200 status, eliminate redirects and duplicate content to save crawl budget, and submit just the sitemap-index file to Google and Bing.

    You can check the link status by uploading or entering the sitemap index file URL in Screaming Frog.
    {{ DiscussionBoard.errors[11074104].message }}
    • Profile picture of the author drneville
      But as far as you know, there is no reason to not submit all of the URLs at once?
      {{ DiscussionBoard.errors[11074134].message }}
      • Profile picture of the author waterguy
        If the pages are unique and valuable then no problem, if they are thin content pages then don't bother. But the way you submit them all is submit the index file URL via Google Search Console. Google will crawl it and add the other individual xml sitemap files. Then it could weeks for Google to crawl all the HTML pages.
        {{ DiscussionBoard.errors[11074223].message }}
Avatar of Unregistered

Trending Topics

  • 11 {{ upvoteCount | shortNum }}

    Free software that can record more that 10 minutes ?

    spartan14 in Internet Marketing

    Hy all i am more a youtube guy and i plan to make a course for my subscribers with some video series and i need to record videos longer that ... [read more]

  • 4 {{ upvoteCount | shortNum }}

    troubleshooting sales

    sparky63 in Internet Marketing

    hi peeps i have just created my first website about premium plr ebook course where you can rebrand and sell on..now i have had quite abit of traffic over the ... [read more]

  • 8 {{ upvoteCount | shortNum }}

    How Do Highly Successful People Stay Positive?

    Hi Gang, I just stumbled upon this great article and wanted to share! 6 Ways Highly Successful People Stay Positive So the question begs, how do YOU stay positive? What ... [read more]

  • 6 {{ upvoteCount | shortNum }}

    What's the deal with Stop words?

    mikehende in SEO

    Hey guys, I setup Yoast SEO plugin on a site and filling in the page title and focus keyword fields I am seeing this: Stop words (a, the, of, for) ... [read more]

  • 218 {{ upvoteCount | shortNum }}

    What is the main reason for the 95% failure rate in internet marketing?

    Here is my take on the matter: Look, a lot of people get into the internet marketing business only because of its huge money-making potential without having a love for ... [read more]