Why did google stop indexing only my Blog pages?

1 year ago

Google tries to determine a limit to the number of requests (which can be quoted as a per second figure) it is willing to make, which in theory is determined to prevent Google overloading the server. It tries to determine the 'capacity' of the server automatically, taking into account previous crawls, it incorporates both activity for the singular hostname, but also other crawls to the same servers for other hostnames.
[promotional link removed by moderator]

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791625].message }}

pssajit

1 year ago

Try manually indexing it using GSC URL Inspection. Manual indexing should work within 24 hours. If it does not index after a day, please add more content to the articles or pages and again do a manual indexing.

[ 1 ] Thanks

Signature

Our World of Dogs

Dog Boarding Bangalore

{{ DiscussionBoard.errors[11791630].message }}

mikehende

1 year ago

Thanks guys but it is showing that the pages were crawled just not indexed. So 3 questions please.

Since the pages were crawled then what's preventing the indexing itself?

Also, what do blogging sites with lots of blogs and churning out new blogs on a regular basis do about this?

Thanks

{{ DiscussionBoard.errors[11791650].message }}

OnlineProxy

1 year ago

Ensuring a clean site structure, using XML sitemaps, fixing crawl errors, and optimizing page speed can help. Check it.

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791660].message }}

Mark Skog

1 year ago

Google has made significant changes to its algorithms. Please go through the Google's guidelines and make sure you're not violating any of it. If your content is of low-quality, it will likely get de-indexed. Many small bloggers have been hit hard by the latest Google update.

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791680].message }}

mikehende

1 year ago

So for confirmation, since google says it requires 'fresh' usable and relevant content a person can churn out as many blogs as desired just the content cannot be duplicate or low quality? And the crawl limit will NOT apply?

Thanks
1 reply

{{ DiscussionBoard.errors[11791703].message }}

Mark Skog 1 year ago

As long as the content is high-quality, you'll be fine even if it is generated by A.I.
- [ 1 ] Thanks
{{ DiscussionBoard.errors[11791710].message }}

ra42

1 year ago

Google will stop crawling sections of a website if it thinks all the pages have similar thin content, or content taken from another site.

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791706].message }}

mikehende

1 year ago

Thanks guys, so what should I do here? I have many blogs with the same old re-written content for the different keywords and topics which is what I am thinking had caused this problem and not crawling budget.

So I am now thinking to remove all re-written blogs and leave only the original blog for each topic, might that be a good idea please?.

Thanks

{{ DiscussionBoard.errors[11791719].message }}

savidge4

1 year ago

I have through this thread a couple times today... kinda at a loss of sorts. lets start with this, so you can better understand this start with reading this: ( https://developers.google.com/search...g-crawl-budget ) forget he said and she said... this is what GOOGLE says.

My business... doing many things builds and hosts websites... more than hundreds. At no point in my career have I ever stated to a client your pages are not indexed because of "crawl budget"

If this is truly an issue you are having, and I have my doubts, this is specifically a hosting issue. Again read the article above, and the whole process is surrounded around the idea of the "Server" and its abilities to handle the queries - this would be specifically your HOST servers ability.

The things you need to make sure... you have an updated sitemap... I would check the robots txt file to ensure nothing is blocked there... What is your overall server capacity and how close to max'd out are you? 80% of max is where things can go downhill, i prefer something like 40% of max for optimal performance.

The answer that was povided to you is your host provider better than suggesting to you that hey our servers are maxed out and there is nothing we can do to help you. The answer is moving to a dedicated server... or a bit more drastic, changing hosts.

And again the link above has many links to different tests and further reading to help you specifically identify and issue you may be having....

I hope that Helps!

[ 1 ] Thanks
1 reply

Signature

Success is an ACT not an idea

{{ DiscussionBoard.errors[11791753].message }}

mikehende

1 year ago

Originally Posted by savidge4

The things you need to make sure... you have an updated sitemap... I would check the robots txt file to ensure nothing is blocked there... What is your overall server capacity and how close to max'd out are you? 80% of max is where things can go downhill, i prefer something like 40% of max for optimal performance.

I hope that Helps!

Yes that certainly helps too Savidge4, thank you for that article which the Indexing guy says he was not aware of the 'crawl health' issue so we are looking into it. He confirmed yesterday that "Sitemap is okay and robots txt is not blocking those urls so you can rule out those". So I will look into the Server account issue and report back.

But guys, my gut feeling though is not the amount of blogs but rather duplicate content and 'stale' content issue. See, we have been doing one monthly blog for 3-4years now re-writing the same content over and over so I feel very confident that this should be the main culprit. Or maybe a combination of both crawl budget and duplicate content issues.

What is confusing guys, is looking at the top competition in this niche who are always at the top of google, their websites have little content and what little content they have it's not as optimized an ours, no frequent blogging or fresh info so why are they always on top?

Thanks
1 reply

{{ DiscussionBoard.errors[11791855].message }}

savidge4

1 year ago

Originally Posted by mikehende

What is confusing guys, is looking at the top competition in this niche who are always at the top of google, their websites have little content and what little content they have it's not as optimized an ours, no frequent blogging or fresh info so why are they always on top?

Because we are talking about "Local Search here" there is primarily one variable that can stand out, and another that doesn't hurt, but doesn't help as much as the first.

"Citations" in all of the proper places and then some can do some magical things when it comes to Local Search

"Back links" can help as well but I would suggest there is more weight in local search with "Citations".

And to clarify the differences... a backlink is just that a link on another page / website that point to a specific page o your website. and a Citation is your NAP ( name address phone number ) information that generally does not have a link ( but a link can be there )

Citations range from a YELP listing to your NAP information in a newspaper article to directory listings - more value is placed on this type if the directory itself is specific in context to what the company does. The holy grail of this type of listing is ones that are curated and served locally - Google is very much local centric. so business listings on a local TV news Site, or listings on a local internet providers site or someone that has a local business directory up and running.

I do have to question the content issue.... one blog post a month, and they are all kind of the same? oh my.... Winter checklists? like disconnecting the hose from the faucet... does the outside faucet have a shut off? freeze valves etc etc.

Did you know that for who knows what reason washer water lines bust more in the summer and generally are oonly good for 5 yrs?

I would start looking for some plumbing newsletter to subscribe to to change up the content type " plumbing newsletter " in Google and you will have article inspiration for days!

[ 1 ] Thanks
1 reply

Signature

Success is an ACT not an idea

{{ DiscussionBoard.errors[11791882].message }}

mikehende

1 year ago

Originally Posted by savidge4

I do have to question the content issue.... one blog post a month, and they are all kind of the same? oh my.... Winter checklists? like disconnecting the hose from the faucet... does the outside faucet have a shut off? freeze valves etc etc.

Did you know that for who knows what reason washer water lines bust more in the summer and generally are oonly good for 5 yrs?

I would start looking for some plumbing newsletter to subscribe to to change up the content type " plumbing newsletter " in Google and you will have article inspiration for days!

Thanks! Yes, we would write one major kw and write a blog about that then every few months would re-write that same blog and so on for each kw. So after a few years there is like maybe 5 or 6 blogs with the same content all just re-written trying to make it look as unique as possible but there's only so many ways one can re-write a piece of content. Also only not much different content available for some kw's.

Whether you believe this or not early this morning I was thinking the same exact thing about how to come up with unique content in a field that's over-run with the same info? You have addressed that too, thank you!

But here's the main problem why I had to do as I have done re-writing the same content. The Plumber has always shot down the idea of writing 'informational' and trending content even though I had warned him a few times in the past that google says it wants both unique and fresh content, catering to the reader. Guy's exact words to me was 'I disagree with google'.

So I had to write the blogs based more around 'his services' than providing industry news or trending topics.

Thanks
1 reply

{{ DiscussionBoard.errors[11791911].message }}

Monetize

1 year ago

Originally Posted by mikehende

Thanks! Yes, we would write one major kw and write a blog about that then every few months would re-write that same blog and so on for each kw. So after a few years there is like maybe 5 or 6 blogs with the same content all just re-written trying to make it look as unique as possible but there's only so many ways one can re-write a piece of content. Also only not much different content available for some kw's.

Whether you believe this or not early this morning I was thinking the same exact thing about how to come up with unique in field that's over-run with the same info? You have addressed that too, thank you!

But here's the main problem why I had to do as I have done re-writing the same content. The Plumber has always shot down the idea of writing 'informational' and trending content even though I had warned him a few times in the past that google says it wants both unique and fresh content, catering to the reader. Guy's exact words to me was 'I disagree with google'.

So I had to write the blogs based more around 'his services' than providing industry news or trending topics.

If the client is not taking your advice, why are you
concerned about the performance of his website.

If I were you, I wouldn't lose any sleep about this
and would probably move on to another client(s).

[ 1 ] Thanks
1 reply

{{ DiscussionBoard.errors[11791913].message }}

mikehende

1 year ago

Originally Posted by Monetize

If the client is not taking your advice, why are you
concerned about the performance of his website.

If I were you, I wouldn't lose any sleep about this
and would probably move on to another client(s).

Yes, I have already told him I won't be doing this much longer.

But for my own knowledge I would like to know for sure what caused it and then to see it working again so I will have that experience under my belt.

Thanks

{{ DiscussionBoard.errors[11791922].message }}

Cricket Mood News

1 year ago

Hey, The same issue persists with our website. For the past 1.5 years, we have been posting news, articles, intresting content, and we have around 150,000 followers on social media platforms including Instagram, Facebook, Twitter, and Telegram etc. However, our pages are still not getting indexed. We hope that one day they will be indexed.

[ 1 ] Thanks
1 reply

{{ DiscussionBoard.errors[11791756].message }}

mikehende

1 year ago

Originally Posted by Cricket Mood News

Hey, The same issue persists with our website. For the past 1.5 years, we have been posting news, articles, intresting content, and we have around 150,000 followers on social media platforms including Instagram, Facebook, Twitter, and Telegram etc. However, our pages are still not getting indexed. We hope that one day they will be indexed.

Hey, can I trouble you to tell if you were like me re-writing existing content or can you say that every blog of yours was unique please?

Thanks

{{ DiscussionBoard.errors[11791923].message }}

vikeshsdp

Banned 1 year ago

[DELETED]

{{ DiscussionBoard.errors[11791762].message }}

Mark Skog 1 year ago

Did you mean the Rank Math plugin for Wordpress?
- [ 1 ] Thanks
{{ DiscussionBoard.errors[11791777].message }}

DonnaLahti

1 year ago

Hey! Google might stop indexing your blog pages if they're having trouble crawling them, or if they find low-quality or duplicate content. It's like Google's saying, "This ain't cool enough for me!" So, make sure your content is unique, high-quality, and crawlable.

[ 1 ] Thanks

Signature

Best Compliance Management Software

{{ DiscussionBoard.errors[11791782].message }}

gogogoing

1 year ago

It happens , if Google thinks all the pages have similar thin content, or content taken from another site.

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791893].message }}

kevingoodson

1 year ago

Yes, as of my experience, I use Blogger for my news site. AI content which I used for the first blog post. Its length is good and also made changes in the AI content, but it is not indexed when I check. For the next post, I take content from the content writer, then I post on the blog. Still have not indexed my second post. Then I understand. I update the whole content of both posts, then submit the URL in URL inspection in the search console. After 48 over I check whether it is indexed or not. Yes, this time it indexes both posts.

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791948].message }}

Software Shop

1 year ago

Low-quality backlinks could negatively affect SEO efforts.
It should be a problem for you.

[ 1 ] Thanks

Signature

Activation license keys - Software shop store

{{ DiscussionBoard.errors[11791962].message }}

mikehende

1 year ago

To update guys, google tech has just confirmed that it is indeed crawl budget issues and says nothing 'they' can do. So they have linked me to an article and have recommended we do all that's mentioned.

Over the next few days I will try to find and remove all duplicate posts.

Thanks
1 reply

{{ DiscussionBoard.errors[11791986].message }}

savidge4

1 year ago

Originally Posted by mikehende

To update guys, google tech has just confirmed that it is indeed crawl budget issues and says nothing 'they' can do. So they have linked me to an article and have recommended we do all that's mentioned.

Over the next few days I will try to find and remove all duplicate posts.

can you share the link to the article?

[ 1 ] Thanks
1 reply

Signature

Success is an ACT not an idea

{{ DiscussionBoard.errors[11791987].message }}

mikehende

1 year ago

I had wanted to do so but in the past my links on this forum have been removed which is the only reason why I did not it but if there is any specific way you advise me on please let me know?

Thanks
1 reply

{{ DiscussionBoard.errors[11791992].message }}

Monetize

1 year ago

Originally Posted by mikehende

I had wanted to do so but in the past my links on this forum have been removed which is the only reason why I did not it but if there is any specific way you advise me on please let me know?

We cannot place promotional, affiliate or commercial
links, informative links are fine, like to news articles
and the one from Goog that we would like to see.

Thanks

{{ DiscussionBoard.errors[11791993].message }}

mikehende

1 year ago

ok then:

https://aioseo.com/crawl-budget/

[ 3 ] Thanks
2 replies

{{ DiscussionBoard.errors[11791994].message }}

DWolfe

1 year ago

Originally Posted by mikehende

ok then:

Wonder how credible this person is - "Kato is one of those rare unicorns born with a pen in his hand--so the legend says. He's authored 2 books and produced content for hundreds of brands" Or just someone trying to sell you a service.

In his text - "Google doesn't publicly share the full criteria used to determine crawl budgets"

Does Google mention Crawl Budgets anywhere? I'm not an SEO expert but most of this thread has been the blind leading the blind.

Thanks
2 replies

{{ DiscussionBoard.errors[11791996].message }}

Monetize

1 year ago

Originally Posted by DWolfe

Wonder how credible this person is - "Kato is one of those rare unicorns born with a pen in his hand--so the legend says. He's authored 2 books and produced content for hundreds of brands" Or just someone trying to sell you a service.

In his text - "Google doesn't publicly share the full criteria used to determine crawl budgets"

Does Google mention Crawl Budgets anywhere? I'm not an SEO expert but most of this thread has been the blind leading the blind.

Apparently, crawl budget is an actual thing: https://developers.google.com/search...g-crawl-budget

[ 1 ] Thanks

{{ DiscussionBoard.errors[11791998].message }}

mikehende 1 year ago

Originally Posted by DWolfe

Does Google mention Crawl Budgets anywhere? I'm not an SEO expert but most of this thread has been the blind leading the blind.

https://developers.google.com/search...g-crawl-budget
- [ 1 ] Thanks
{{ DiscussionBoard.errors[11792000].message }}

savidge4 1 year ago

Originally Posted by mikehende

ok then:

https://aioseo.com/crawl-budget/

I would say not the best resource. The definitions and limits use the term "Site" and that specifically is not the case ( well in most instances - you could technically have a site parked on a dedicated server ) The primary restraining factor is "Server" speed. Slower speed, less bot visits, faster speeds - more bot visits.

When we are talking about servers... we are talking about the storage space you have your site hosted on... so say bluehost. If you have dedicated hosting, you stand the chances of this problem happening to be more server error issues than anything. Shared hosting, and this is where there can be issues out of your control.

So shared hosting with have say 16 websites allocated on a server. In regards to Googlebot the servers Crawl Budget will be only as good as the worst performing site of the 16. meaning if you happen to be sharing hosting with a site that has a gazilion site server errors... the entire 16 sites will have a piss poor server budget. And aside from changing hosts, or hosting plans to say "dedicated" there is absolutely nothing you can do.

I have been in this situation before... on shared hosting, and it was next to impossible to get content indexed, and had other sites - on other servers, and content was crawled and indexed in minutes to hours.

running all of the URL test and checking for page errors etc, give you an idea of the health of your site... if everything checks out and looks fine, then you are left with the scenario of shared hosting and the crawl budget is determined by the weakest link.
- [ 1 ] Thanks
Signature

Success is an ACT not an idea
{{ DiscussionBoard.errors[11792331].message }}

Alora Carl

1 year ago

I can share the unique way without feteching from the google search console. Add the internal link of those pages that you want to index (but this page have some organic traffic). When google will crawl it they will automatically index all.

[ 1 ] Thanks

{{ DiscussionBoard.errors[11792289].message }}

Tootsie68

1 year ago

[DELETED]

{{ DiscussionBoard.errors[11792646].message }}

DABK 1 year ago

Mike.
Something to keep in mind: Google has to crawl a page to know it duplicates another page or has thin content.

Once it crawls, it decides what to do about it. One of the options is index/no index. If it decides to index, then it needs to decide where and for what keywords to show it.

It does not crawl pages because it cannot find them/ get to them.

Easiest way I know to get a page crawled is to go to your Google webmaster master account (search console) and submit a request to have the page crawled.

It is best, however, if you are on a server that does not have issues. I like site ground.

As regards content, your client's approach is them getting you to shoot them in the foot.

Their business is local. Their clients are people. People are less interested in general plumbing information and far more about them and their clients.

A "this is how you drain a washing machine properly" is interesting to the few who want to do it themselves. "Day before yesterday, I got a call from a guy in Queens. His toilet was leaking. I fixed that in 5 minutes. But let me tell you how this toilet got to be leaking... Or: it was an easy job, once I got past his parakeet..." Those are far more interesting, you can place all kinds of general plumbing info in these stories and they come across as duplicates less often... One guy has a parakeet, the next one has a mother-in-law who flashes you, the next one a father who works for Honda, makes the a/c knobs...
- [ 1 ] Thanks
{{ DiscussionBoard.errors[11792666].message }}
DWolfe 1 year ago

Originally Posted by Tootsie68

Google is having trouble adding new blog posts to a website.

Where did you get that information from? If a WordPress site is on a major hosting site like Hostgator, Site Ground, Blue Host, etc. Why would Google be adding Blog Posts, please explain.
- Thanks
{{ DiscussionBoard.errors[11792681].message }}

mikehende

1 year ago

Hey guys, just updating. I had removed all duplicate posts, 32 of them and the original posts have gotten indexed [3 left] but calls have started coming again, just wanted to update and to say thanks to everyone for your help!

[ 2 ] Thanks

{{ DiscussionBoard.errors[11792905].message }}

feronenab

1 year ago

Crawl budget can affect how quickly Google indexes your pages. To improve indexing in the future, нour site's architecture need clean, quality content.