39 {{ upvoteCount | shortNum }}
39 {{ upvoteCount | shortNum }}

Why did google stop indexing only my Blog pages?

by mikehende
33 replies
  • SEO
    • |
Hey guys, a local plumber here has a site which was ranking very well for the past 5 years until last year when his Stats started slowly dropping each month. We were baffled by this as we were consistently still doing one monthly blog and monthly backlinking without missing not even month.

I only recently noticed that only his blog pages had issues all this time not being indexed so my fault in not seeing this sooner. GSC shows 59 blog pages not indexed so I reached out to a WP Coder who works on getting those pages indexed. Here's what he says when I asked 'why the pages were not being indexed any more'?

Issue is crawling. And also a crawl budget. Crawl budget is the number of pages Googlebot crawls and indexes on a website within a given timeframe.
So if your number of pages exceeds your site's crawl budget, you're going to have pages on your site that aren't indexed. and they won't get indexed until the crawl budget is reset.
So I then asked what needs to be done to 'Reset the crawl budget' and also if that will solve the problem so no more indexing issues? His response:

Nothing.. we can't do anything to it..They pile up over time, we can try to index them. Most of These urls have not been crawled properly so i will re-crawl them which will result in Indexing. My work only affect current pages. So there won't be any fix for future post or pages.
But if you have a lot of urls already indexed those will help new one get indexed and also crawler won't have to worry about old urls..
So guys, any word on all of this please, is he 100% accurate in every he has said that nothing can be done to make sure all 'future' pages will be indexed?
#blog #google #indexing #pages #stop
Avatar of Unregistered
  • Profile picture of the author inhubsa
    Google tries to determine a limit to the number of requests (which can be quoted as a per second figure) it is willing to make, which in theory is determined to prevent Google overloading the server. It tries to determine the 'capacity' of the server automatically, taking into account previous crawls, it incorporates both activity for the singular hostname, but also other crawls to the same servers for other hostnames.
    [promotional link removed by moderator]
    {{ DiscussionBoard.errors[11791625].message }}
  • Profile picture of the author pssajit
    Try manually indexing it using GSC URL Inspection. Manual indexing should work within 24 hours. If it does not index after a day, please add more content to the articles or pages and again do a manual indexing.
    Signature

    Our World of Dogs

    Dog Boarding Bangalore

    {{ DiscussionBoard.errors[11791630].message }}
  • Profile picture of the author mikehende
    Thanks guys but it is showing that the pages were crawled just not indexed. So 3 questions please.

    Since the pages were crawled then what's preventing the indexing itself?

    Also, what do blogging sites with lots of blogs and churning out new blogs on a regular basis do about this?
    {{ DiscussionBoard.errors[11791650].message }}
  • Profile picture of the author OnlineProxy
    Ensuring a clean site structure, using XML sitemaps, fixing crawl errors, and optimizing page speed can help. Check it.
    {{ DiscussionBoard.errors[11791660].message }}
  • Profile picture of the author Mark Skog
    Google has made significant changes to its algorithms. Please go through the Google's guidelines and make sure you're not violating any of it. If your content is of low-quality, it will likely get de-indexed. Many small bloggers have been hit hard by the latest Google update.
    {{ DiscussionBoard.errors[11791680].message }}
  • Profile picture of the author mikehende
    So for confirmation, since google says it requires 'fresh' usable and relevant content a person can churn out as many blogs as desired just the content cannot be duplicate or low quality? And the crawl limit will NOT apply?
    {{ DiscussionBoard.errors[11791703].message }}
    • Profile picture of the author Mark Skog
      As long as the content is high-quality, you'll be fine even if it is generated by A.I.
      {{ DiscussionBoard.errors[11791710].message }}
  • Profile picture of the author ra42
    Google will stop crawling sections of a website if it thinks all the pages have similar thin content, or content taken from another site.
    {{ DiscussionBoard.errors[11791706].message }}
  • Profile picture of the author mikehende
    Thanks guys, so what should I do here? I have many blogs with the same old re-written content for the different keywords and topics which is what I am thinking had caused this problem and not crawling budget.

    So I am now thinking to remove all re-written blogs and leave only the original blog for each topic, might that be a good idea please?.
    {{ DiscussionBoard.errors[11791719].message }}
  • Profile picture of the author savidge4
    I have through this thread a couple times today... kinda at a loss of sorts. lets start with this, so you can better understand this start with reading this: ( https://developers.google.com/search...g-crawl-budget ) forget he said and she said... this is what GOOGLE says.

    My business... doing many things builds and hosts websites... more than hundreds. At no point in my career have I ever stated to a client your pages are not indexed because of "crawl budget"

    If this is truly an issue you are having, and I have my doubts, this is specifically a hosting issue. Again read the article above, and the whole process is surrounded around the idea of the "Server" and its abilities to handle the queries - this would be specifically your HOST servers ability.

    The things you need to make sure... you have an updated sitemap... I would check the robots txt file to ensure nothing is blocked there... What is your overall server capacity and how close to max'd out are you? 80% of max is where things can go downhill, i prefer something like 40% of max for optimal performance.

    The answer that was povided to you is your host provider better than suggesting to you that hey our servers are maxed out and there is nothing we can do to help you. The answer is moving to a dedicated server... or a bit more drastic, changing hosts.

    And again the link above has many links to different tests and further reading to help you specifically identify and issue you may be having....

    I hope that Helps!
    Signature
    Success is an ACT not an idea
    {{ DiscussionBoard.errors[11791753].message }}
    • Profile picture of the author mikehende
      Originally Posted by savidge4 View Post

      The things you need to make sure... you have an updated sitemap... I would check the robots txt file to ensure nothing is blocked there... What is your overall server capacity and how close to max'd out are you? 80% of max is where things can go downhill, i prefer something like 40% of max for optimal performance.

      I hope that Helps!
      Yes that certainly helps too Savidge4, thank you for that article which the Indexing guy says he was not aware of the 'crawl health' issue so we are looking into it. He confirmed yesterday that "Sitemap is okay and robots txt is not blocking those urls so you can rule out those". So I will look into the Server account issue and report back.

      But guys, my gut feeling though is not the amount of blogs but rather duplicate content and 'stale' content issue. See, we have been doing one monthly blog for 3-4years now re-writing the same content over and over so I feel very confident that this should be the main culprit. Or maybe a combination of both crawl budget and duplicate content issues.

      What is confusing guys, is looking at the top competition in this niche who are always at the top of google, their websites have little content and what little content they have it's not as optimized an ours, no frequent blogging or fresh info so why are they always on top?
      {{ DiscussionBoard.errors[11791855].message }}
      • Profile picture of the author savidge4
        Originally Posted by mikehende View Post

        What is confusing guys, is looking at the top competition in this niche who are always at the top of google, their websites have little content and what little content they have it's not as optimized an ours, no frequent blogging or fresh info so why are they always on top?
        Because we are talking about "Local Search here" there is primarily one variable that can stand out, and another that doesn't hurt, but doesn't help as much as the first.

        "Citations" in all of the proper places and then some can do some magical things when it comes to Local Search

        "Back links" can help as well but I would suggest there is more weight in local search with "Citations".

        And to clarify the differences... a backlink is just that a link on another page / website that point to a specific page o your website. and a Citation is your NAP ( name address phone number ) information that generally does not have a link ( but a link can be there )

        Citations range from a YELP listing to your NAP information in a newspaper article to directory listings - more value is placed on this type if the directory itself is specific in context to what the company does. The holy grail of this type of listing is ones that are curated and served locally - Google is very much local centric. so business listings on a local TV news Site, or listings on a local internet providers site or someone that has a local business directory up and running.

        I do have to question the content issue.... one blog post a month, and they are all kind of the same? oh my.... Winter checklists? like disconnecting the hose from the faucet... does the outside faucet have a shut off? freeze valves etc etc.

        Did you know that for who knows what reason washer water lines bust more in the summer and generally are oonly good for 5 yrs?

        I would start looking for some plumbing newsletter to subscribe to to change up the content type " plumbing newsletter " in Google and you will have article inspiration for days!
        Signature
        Success is an ACT not an idea
        {{ DiscussionBoard.errors[11791882].message }}
        • Profile picture of the author mikehende
          Originally Posted by savidge4 View Post

          I do have to question the content issue.... one blog post a month, and they are all kind of the same? oh my.... Winter checklists? like disconnecting the hose from the faucet... does the outside faucet have a shut off? freeze valves etc etc.

          Did you know that for who knows what reason washer water lines bust more in the summer and generally are oonly good for 5 yrs?

          I would start looking for some plumbing newsletter to subscribe to to change up the content type " plumbing newsletter " in Google and you will have article inspiration for days!
          Thanks! Yes, we would write one major kw and write a blog about that then every few months would re-write that same blog and so on for each kw. So after a few years there is like maybe 5 or 6 blogs with the same content all just re-written trying to make it look as unique as possible but there's only so many ways one can re-write a piece of content. Also only not much different content available for some kw's.

          Whether you believe this or not early this morning I was thinking the same exact thing about how to come up with unique content in a field that's over-run with the same info? You have addressed that too, thank you!

          But here's the main problem why I had to do as I have done re-writing the same content. The Plumber has always shot down the idea of writing 'informational' and trending content even though I had warned him a few times in the past that google says it wants both unique and fresh content, catering to the reader. Guy's exact words to me was 'I disagree with google'.

          So I had to write the blogs based more around 'his services' than providing industry news or trending topics.
          {{ DiscussionBoard.errors[11791911].message }}
          • Profile picture of the author Monetize
            Originally Posted by mikehende View Post

            Thanks! Yes, we would write one major kw and write a blog about that then every few months would re-write that same blog and so on for each kw. So after a few years there is like maybe 5 or 6 blogs with the same content all just re-written trying to make it look as unique as possible but there's only so many ways one can re-write a piece of content. Also only not much different content available for some kw's.

            Whether you believe this or not early this morning I was thinking the same exact thing about how to come up with unique in field that's over-run with the same info? You have addressed that too, thank you!

            But here's the main problem why I had to do as I have done re-writing the same content. The Plumber has always shot down the idea of writing 'informational' and trending content even though I had warned him a few times in the past that google says it wants both unique and fresh content, catering to the reader. Guy's exact words to me was 'I disagree with google'.

            So I had to write the blogs based more around 'his services' than providing industry news or trending topics.

            If the client is not taking your advice, why are you
            concerned about the performance of his website.

            If I were you, I wouldn't lose any sleep about this
            and would probably move on to another client(s).
            {{ DiscussionBoard.errors[11791913].message }}
            • Profile picture of the author mikehende
              Originally Posted by Monetize View Post

              If the client is not taking your advice, why are you
              concerned about the performance of his website.

              If I were you, I wouldn't lose any sleep about this
              and would probably move on to another client(s).
              Yes, I have already told him I won't be doing this much longer.

              But for my own knowledge I would like to know for sure what caused it and then to see it working again so I will have that experience under my belt.
              {{ DiscussionBoard.errors[11791922].message }}
  • Profile picture of the author Cricket Mood News
    Hey, The same issue persists with our website. For the past 1.5 years, we have been posting news, articles, intresting content, and we have around 150,000 followers on social media platforms including Instagram, Facebook, Twitter, and Telegram etc. However, our pages are still not getting indexed. We hope that one day they will be indexed.
    {{ DiscussionBoard.errors[11791756].message }}
    • Profile picture of the author mikehende
      Originally Posted by Cricket Mood News View Post

      Hey, The same issue persists with our website. For the past 1.5 years, we have been posting news, articles, intresting content, and we have around 150,000 followers on social media platforms including Instagram, Facebook, Twitter, and Telegram etc. However, our pages are still not getting indexed. We hope that one day they will be indexed.
      Hey, can I trouble you to tell if you were like me re-writing existing content or can you say that every blog of yours was unique please?
      {{ DiscussionBoard.errors[11791923].message }}
  • Profile picture of the author vikeshsdp
    You can use the ranking math API to force indexing. It is very useful.
    {{ DiscussionBoard.errors[11791762].message }}
  • Profile picture of the author DonnaLahti
    Hey! Google might stop indexing your blog pages if they're having trouble crawling them, or if they find low-quality or duplicate content. It's like Google's saying, "This ain't cool enough for me!" So, make sure your content is unique, high-quality, and crawlable.
    Signature
    Best Compliance Management Software
    {{ DiscussionBoard.errors[11791782].message }}
  • Profile picture of the author gogogoing
    It happens , if Google thinks all the pages have similar thin content, or content taken from another site.
    {{ DiscussionBoard.errors[11791893].message }}
  • Profile picture of the author kevingoodson
    Yes, as of my experience, I use Blogger for my news site. AI content which I used for the first blog post. Its length is good and also made changes in the AI content, but it is not indexed when I check. For the next post, I take content from the content writer, then I post on the blog. Still have not indexed my second post. Then I understand. I update the whole content of both posts, then submit the URL in URL inspection in the search console. After 48 over I check whether it is indexed or not. Yes, this time it indexes both posts.
    {{ DiscussionBoard.errors[11791948].message }}
  • Profile picture of the author Software Shop
    Low-quality backlinks could negatively affect SEO efforts.
    It should be a problem for you.
    Signature
    Activation license keys - Software shop store
    {{ DiscussionBoard.errors[11791962].message }}
  • Profile picture of the author mikehende
    To update guys, google tech has just confirmed that it is indeed crawl budget issues and says nothing 'they' can do. So they have linked me to an article and have recommended we do all that's mentioned.

    Over the next few days I will try to find and remove all duplicate posts.
    {{ DiscussionBoard.errors[11791986].message }}
    • Profile picture of the author savidge4
      Originally Posted by mikehende View Post

      To update guys, google tech has just confirmed that it is indeed crawl budget issues and says nothing 'they' can do. So they have linked me to an article and have recommended we do all that's mentioned.

      Over the next few days I will try to find and remove all duplicate posts.
      can you share the link to the article?
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[11791987].message }}
      • Profile picture of the author mikehende
        I had wanted to do so but in the past my links on this forum have been removed which is the only reason why I did not it but if there is any specific way you advise me on please let me know?
        {{ DiscussionBoard.errors[11791992].message }}
        • Profile picture of the author Monetize
          Originally Posted by mikehende View Post

          I had wanted to do so but in the past my links on this forum have been removed which is the only reason why I did not it but if there is any specific way you advise me on please let me know?
          We cannot place promotional, affiliate or commercial
          links, informative links are fine, like to news articles
          and the one from Goog that we would like to see.
          {{ DiscussionBoard.errors[11791993].message }}
  • Profile picture of the author mikehende
    ok then:

    https://aioseo.com/crawl-budget/
    {{ DiscussionBoard.errors[11791994].message }}
    • Profile picture of the author DWolfe
      Originally Posted by mikehende View Post

      ok then:
      Wonder how credible this person is - "Kato is one of those rare unicorns born with a pen in his hand--so the legend says. He's authored 2 books and produced content for hundreds of brands" Or just someone trying to sell you a service.

      In his text - "Google doesn't publicly share the full criteria used to determine crawl budgets"

      Does Google mention Crawl Budgets anywhere? I'm not an SEO expert but most of this thread has been the blind leading the blind.
      {{ DiscussionBoard.errors[11791996].message }}
      • Profile picture of the author Monetize
        Originally Posted by DWolfe View Post

        Wonder how credible this person is - "Kato is one of those rare unicorns born with a pen in his hand--so the legend says. He's authored 2 books and produced content for hundreds of brands" Or just someone trying to sell you a service.

        In his text - "Google doesn't publicly share the full criteria used to determine crawl budgets"

        Does Google mention Crawl Budgets anywhere? I'm not an SEO expert but most of this thread has been the blind leading the blind.

        Apparently, crawl budget is an actual thing: https://developers.google.com/search...g-crawl-budget
        {{ DiscussionBoard.errors[11791998].message }}
      • Profile picture of the author mikehende
        Originally Posted by DWolfe View Post

        Does Google mention Crawl Budgets anywhere? I'm not an SEO expert but most of this thread has been the blind leading the blind.
        https://developers.google.com/search...g-crawl-budget
        {{ DiscussionBoard.errors[11792000].message }}
    • Profile picture of the author savidge4
      Originally Posted by mikehende View Post

      I would say not the best resource. The definitions and limits use the term "Site" and that specifically is not the case ( well in most instances - you could technically have a site parked on a dedicated server ) The primary restraining factor is "Server" speed. Slower speed, less bot visits, faster speeds - more bot visits.

      When we are talking about servers... we are talking about the storage space you have your site hosted on... so say bluehost. If you have dedicated hosting, you stand the chances of this problem happening to be more server error issues than anything. Shared hosting, and this is where there can be issues out of your control.

      So shared hosting with have say 16 websites allocated on a server. In regards to Googlebot the servers Crawl Budget will be only as good as the worst performing site of the 16. meaning if you happen to be sharing hosting with a site that has a gazilion site server errors... the entire 16 sites will have a piss poor server budget. And aside from changing hosts, or hosting plans to say "dedicated" there is absolutely nothing you can do.

      I have been in this situation before... on shared hosting, and it was next to impossible to get content indexed, and had other sites - on other servers, and content was crawled and indexed in minutes to hours.

      running all of the URL test and checking for page errors etc, give you an idea of the health of your site... if everything checks out and looks fine, then you are left with the scenario of shared hosting and the crawl budget is determined by the weakest link.
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[11792331].message }}
  • Profile picture of the author Alora Carl
    I can share the unique way without feteching from the google search console. Add the internal link of those pages that you want to index (but this page have some organic traffic). When google will crawl it they will automatically index all.
    {{ DiscussionBoard.errors[11792289].message }}
Avatar of Unregistered

Trending Topics