Sitemap on WP Blog w/1m entries

8 replies
I have a WP Blog with over 1million posts. I dont think it will be a problem to get the sitemap generated, however I need to know the best way to arrange the SM so it gets indexed as much as it can>

Do i need multiple maps? Is there some helpful tutorial somewhere?
#blog #entries #sitemap #w or 1m
  • Profile picture of the author wayfarer
    Chrissake, why does this topic keep coming up? A sitemap is not going to help you get indexed any faster. Only fresh content matters anyway. You can just ping Google to let them know there is new content. Here is the official statement from Google: Official Google Blog: Got blog? Will ping. Your Wordpress blog has a feature that will allow you to automatically do this.

    Making a sitemap and submitting it every time you post something new technically will help your new post be indexed faster, but it has nothing to do with the sitemap, and is totally wasteful of server resources.
    Signature
    I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
    {{ DiscussionBoard.errors[539834].message }}
    • Profile picture of the author Virtual Banker
      Originally Posted by wayfarer View Post

      Chrissake, why does this topic keep coming up? A sitemap is not going to help you get indexed any faster. Only fresh content matters anyway. You can just ping Google to let them know there is new content. Here is the official statement from Google: Official Google Blog: Got blog? Will ping. Your Wordpress blog has a feature that will allow you to automatically do this.

      Making a sitemap and submitting it every time you post something new technically will help your new post be indexed faster, but it has nothing to do with the sitemap, and is totally wasteful of server resources.
      actually I wasnt looking for a geek flame, but thanks for your input, I was actually looking to see if anybody had experience with large sitemaps and their experience getting them indexed. Of course we do ping, but we would have to ping for about 100 years to get all 1m pinged without a ban .

      Thanks.
      {{ DiscussionBoard.errors[540933].message }}
      • Profile picture of the author aisling
        One of my websites is large, but not that huge. (Mine is only about 500 articles, but it started as a hobby website and I'm the only author of those articles.)

        But, I understand the navigation headaches of a huge and diverse website.

        So, if this is about navigation as much as SEO issues, here's my advice:

        For starters, I use the regular sitemap plugin (Dagon Design Sitemap Generator, Sitemap Generator Plugin for Wordpress Dagon Design).

        Then, I supplement that with Sobek's Posts in Category (http://wordpress.sobek.pl/sobeks-pos...tegory-plugin/ ) and manually generate some mini-sitemaps for specific interests of my website visitors.

        It was more than I needed for my site, so I don't use this any more, but you may also like something like Folding Category List (Folding Category Widget - Lonewolf Online )
        Signature

        WordPress themes made easy: Sites that Soar!

        {{ DiscussionBoard.errors[542103].message }}
  • Profile picture of the author Mike Adams
    Use an expandable/ collapsable sitemap to show your visitors and an .xml sitemap for the search engines
    Signature
    P.S. If you found my post useful, please click the "Thanks" button below...
    {{ DiscussionBoard.errors[543395].message }}
    • Profile picture of the author Bruce Hearder
      A rolling sitemap maybe the best way to go.
      I would suggest keeping the number of enteries in the SM to less than 1000, and every few hours or days (depending on how often the Search Engines (SE) are hitting the site) drop off the first 200-300 posts and add the next 200-300 posts.

      This way the SE keep comoing back more and more frequently and you will get just about everything indexed.

      Note how I said, just "about everything". Thats becuase you will NEVER get every post indexed, it just the way BigG works.
      There seems to be some sort of magic ceiling on number of pages on a site indexed..

      Just gotta keep telling BigG that is there..

      Hope this helps..

      Take care

      Bruce
      {{ DiscussionBoard.errors[544692].message }}
      • Profile picture of the author Virtual Banker
        Originally Posted by Bruce Hearder View Post

        A rolling sitemap maybe the best way to go.
        I would suggest keeping the number of enteries in the SM to less than 1000, and every few hours or days (depending on how often the Search Engines (SE) are hitting the site) drop off the first 200-300 posts and add the next 200-300 posts.

        This way the SE keep comoing back more and more frequently and you will get just about everything indexed.

        Note how I said, just "about everything". Thats becuase you will NEVER get every post indexed, it just the way BigG works.
        There seems to be some sort of magic ceiling on number of pages on a site indexed..

        Just gotta keep telling BigG that is there..

        Hope this helps..

        Take care

        Bruce
        Bruce

        now that is an interesting concept. I do this to a degree with the random posts plugin in my sidebar. (iee. links to random posts are displayed on each load)

        but to do that with a sitemap...Essentially dynamically generate a random sitemap each time it is loaded!!!

        that sounds interesting. Now I could have my devs bang away on that, but maybe there is already a WP plugin that does just this?
        {{ DiscussionBoard.errors[545020].message }}
  • Profile picture of the author wayfarer
    Why would you ping the whole site, or even need a rolling sitemap? Are all million posts going to be constantly updated? My advice, let Google take care of itself. One they index you they will check back periodically. In my opinion, you aren't going to be helping yourself any of this. You should only ping the latest posts. A sitemap isn't even needed.

    There seems to be some sort of magic ceiling on number of pages on a site indexed..
    There is no such limit. This is kind of like saying there is a limit as to how big the internet is allowed to grow. However, if your posts are buried too deeply it is possible they won't be indexed. Pinging or sitemaps will not help this situation. The best thing you can do is build pages that link to the deeper parts of the site, so that every page on the site can be reached in three or so clicks.
    Signature
    I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
    {{ DiscussionBoard.errors[546215].message }}
    • Profile picture of the author Bruce Hearder
      Originally Posted by wayfarer View Post

      There is no such limit.
      Technically you are absolutely correct!

      But when you got a very big site (and I talking BIG), G never seems to index the entire site. It always seems to stop at some point. The number of pages indexed seems to change from site to site, but they NEVER index every page you've got..

      But something stops their system from indexing every page.
      I don't know what the limitation is, but something stops it.

      But we can use this lmitation to our advantage though!

      Everyone should be checking their server logs on a frequent basis to see where their traffic is coming from, and many times you will find that there are links to your site from pages that are not indexed in G or Y! or MSN.

      Part of the problem discussed above. The page is getting traffic but the SE have not indexed the page.

      So, simply ping the page with your link on it( do it at least a 4-5 times over the next day or so) and you might just get the page indexed and you now have another backlink in the SE to your website.

      A slow process but when you hit the "backlink ceiling" (anyone who's into SEO in a serious way will know what i mean), every extra link counts.

      Take care

      Bruce
      {{ DiscussionBoard.errors[546829].message }}

Trending Topics