Submitting large sitemaps to Google

by drneville 6 replies
We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.
#search engine optimization #google #large #sitemaps #submitting
Avatar of Unregistered
  • Profile picture of the author yukon
    Originally Posted by drneville View Post

    We have a relatively new website/domain (about 1.5 years old) and it has ~15 million URLs on it. I was wondering if there is an issue submitting sitemaps containing all of the URLs I'd like to eventually have indexed vs. "spoon feeding" fewer sitemaps over an extended period of time. In my mind, this is simply telling Google: "here's my site and these are the URLs I'd like you to consider indexing." But not sure if showing 15 million all at once creates an issue or if there is a different strategy to employ.



    Don't add 15 million URLs to a single xml file.

    You need to use a sitemap index xml file that includes multiple xml sitemaps, example, each category could have their own xml sitemap.

    Add a master.xml file to your robots.txt file.
    • Sitemap: hxxp://domain.com/sitemap/master.xml

    The master.xml file will contain URLs to each of your category.xml files (Google link example above).
    Signature
    Hi
    {{ DiscussionBoard.errors[11074027].message }}
    • Profile picture of the author drneville
      Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.
      {{ DiscussionBoard.errors[11074051].message }}
      • Profile picture of the author yukon
        Originally Posted by drneville View Post

        Thanks - yes - we have broken them down this way. My question was more around whether we should submit all sitemap index & sitemap files at once. Is there a "penalty" for doing so that inhibits indexing/ranking of our URLs.

        That's ok, Google won't index/crawl all the URLs at one time anyways.
        Signature
        Hi
        {{ DiscussionBoard.errors[11074067].message }}
  • Profile picture of the author waterguy
    It's been a while but there was a stat a few years ago if there are more that 10% broken links in your sitemap the bots will not follow them, so make sure your links all resolve to a 200 status, eliminate redirects and duplicate content to save crawl budget, and submit just the sitemap-index file to Google and Bing.

    You can check the link status by uploading or entering the sitemap index file URL in Screaming Frog.
    {{ DiscussionBoard.errors[11074104].message }}
    • Profile picture of the author drneville
      But as far as you know, there is no reason to not submit all of the URLs at once?
      {{ DiscussionBoard.errors[11074134].message }}
      • Profile picture of the author waterguy
        If the pages are unique and valuable then no problem, if they are thin content pages then don't bother. But the way you submit them all is submit the index file URL via Google Search Console. Google will crawl it and add the other individual xml sitemap files. Then it could weeks for Google to crawl all the HTML pages.
        {{ DiscussionBoard.errors[11074223].message }}
Avatar of Unregistered

Trending Topics

  • 47 {{ upvoteCount | shortNum }}

    Warriors .... Warriors.... what would you like to LEARN?

    Good Morning awesome Warriors... I'm happy to announce that The Warrior Forum is planning to create our first COURSE. Yes... Powered by Warrior Wisdom... yeah!! Made at home. But before ... [read more]

  • 3 {{ upvoteCount | shortNum }}

    I don't want to get another cease and desist letter about my domain name...

    bigcat1967 in Internet Marketing

    I was once threatened by a Lawsuit concerning a domain name that I owned - and I don't want to make the same mistake. So therefore, I want to make ... [read more]

  • 44 {{ upvoteCount | shortNum }}

    How Do Highly Successful People Stay Positive?

    Hi Gang, I just stumbled upon this great article and wanted to share! 6 Ways Highly Successful People Stay Positive So the question begs, how do YOU stay positive? What ... [read more]

  • 2 {{ upvoteCount | shortNum }}

    The website is frozen after 301 redirect

    AdamTear12 in SEO

    Hello mates, I made a 301 redirect on one of my websites. Purchased a new domain, installed WordPress on it, and then did a 301 redirect in order to inject ... [read more]

  • 1 {{ upvoteCount | shortNum }}

    Facebook Group Members Email Extract

    Kamal Aslanov in Social Networks

    Hi guys! I am looking for some tool to export email list of facebook group members. Is there any way? Thanks.