Problems with indexing my inside pages - Google | Warrior Forum - The #1 Digital Marketing Forum & Marketplace

Thank you very much. I just tried it.

It seems that there is a new error since a Freelancer edited my page 2 days before.

Robots.txt fetch failed

You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.

Thanks
1 reply

{{ DiscussionBoard.errors[9818520].message }}

yukon

Banned 9 years ago

Originally Posted by ClaudeMonet

Thank you very much. I just tried it.

It seems that there is a new error since a Freelancer edited my page 2 days before.

Robots.txt fetch failed

You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.

I already checked:

/robots.txt
/sitemap.xml

Both URLs work.

PM me a few of the problem pages/URLs.

[edit]
Make sure you submit robots.txt & not Robots.txt, maybe Fetch as Google is case sensitive on the URL.

Also, your robots.txt doesn't really contain anything for indexing pages, there's nothing pointing to the xml sitemap. Submit that sitemap.xml file to Fetch as Google.

[ 1 ] Thanks
1 reply

{{ DiscussionBoard.errors[9818529].message }}

ClaudeMonet 9 years ago

it is r and not R
- Thanks
- 1 reply
{{ DiscussionBoard.errors[9818536].message }}
- paulgl 9 years ago
  
  It could be a server problem. It happens.
  
  here's a quick test, but you need to be logged into facebook.
  Send it through their debugger.
  https://developers.facebook.com/tools/debug/og/object/
  I plugged in:
  http:// www. redollar.com/start.html
  (without spaces, of course)
  
  It shows errors.
  Scrolling down and clicking on: See exactly what our scraper sees for your URL
  Opens up a new window, and it says, Document returned no data
  
  That means nothing(?) can crawl your site, my guess, server issue.
  
  You don't need a robots.txt
  
  Paul
  
  [ 1 ] Thanks
  
  1 reply
  
  Signature
  
  If you were disappointed in your results today, lower your standards tomorrow.
  
  {{ DiscussionBoard.errors[9818584].message }}
  
  ClaudeMonet 9 years ago
  
  Thank for your efforts.
  
  Do you mean my provider?
  
  What can I do against it?
  
  Thanks
  
  {{ DiscussionBoard.errors[9818726].message }}

yukon

Banned 9 years ago

Need to see the problem URLs.

Try this...

Do a site:domain.com/exact-internal-page-url search to prove the pages aren't indexed in Google SERPs.

Go to Google Webmaster Tools & try submitting a problem page/URL via the Fetch as Google tool. See If Google will index some of the pages/URLs. The Fetch as Google tool is usually fast at indexing pages (few min.).

[ 1 ] Thanks

{{ DiscussionBoard.errors[9818506].message }}

yukon

Banned 9 years ago

Did you get your pages indexed yet?

Thanks
1 reply

{{ DiscussionBoard.errors[9818743].message }}

ClaudeMonet 9 years ago

NO! NOTHING!

maybe I should change the hosting. Now it's godaddy - think about using liquidweb...
- Thanks
- 1 reply
{{ DiscussionBoard.errors[9818755].message }}
- paulgl 9 years ago
  
  Is it a wordpress site? Do you have any recent plugins?
  
  Something has stopped these crawlers in their tracks, either
  something was expected, a different size, image, url, etc. than
  what was thought to be there. Even some sort of authentication error.
  
  I'd go more with server. Did you get some other sort of hosting
  recently? A support contact to your host describing the issues
  might be what you need to do.
  
  Paul
  
  Thanks
  
  2 replies
  
  Signature
  
  If you were disappointed in your results today, lower your standards tomorrow.
  
  {{ DiscussionBoard.errors[9818814].message }}
  
  ClaudeMonet 9 years ago
  
  no wordpress at the moment, just a html template.
  
  Thanks
  
  {{ DiscussionBoard.errors[9818818].message }}
  
  yukon Banned 9 years ago
  
  Originally Posted by ClaudeMonet
  
  NO! NOTHING!
  
  maybe I should change the hosting. Now it's godaddy - think about using liquidweb...
  
  The host isn't the problem.
  
  Send me a couple of the pages that you can't get indexed, I'll take a look to see what's going on. You have 60+ indexed pages out of 100+ pages on your site, I don't have time to verify each individual page but I'll look at a couple of pages If you send me the exact URLs.
  
  Originally Posted by paulgl
  
  Is it a wordpress site? Do you have any recent plugins?
  
  Paul
  
  No, it's HTML.
  
  [ 1 ] Thanks
  
  {{ DiscussionBoard.errors[9818820].message }}

gordongecko

9 years ago

Hello,

You can use XML format files as well for this. Or you can try RSS. Or search for instant XML sitemap online creator in google. Hope it'll help you do index your site. Contact with google support department or read FAQ to learn more.

Thank You

Thanks

Signature

"Iâ€™m stronger because I had to be, smart because of my mistakes, happier because of the sadness Iâ€™ve known, and now wiser because Iâ€™ve learned." - Unknown

{{ DiscussionBoard.errors[9819069].message }}

ahmansoor

9 years ago

Other than checking robots.txt and HTML robots tag, create a sitemap your site, improve other navigations, participate in social media and share links over there.

Thanks
1 reply

Signature

SEO Consultant's Website - SEO Specialist's Blog - SEO Consultant's Twitter

{{ DiscussionBoard.errors[9819078].message }}

paulgl

9 years ago

The fact that a debugger, like the free on at FB shows a

"Document returned no data"

Shows that it is either a server problem or an authentication problem,
which could also be related to a server problem. That is, your
host.

You can access the pages, as well as probably 99% of the viewers.
Browsers are very, very forgiving in 2015.

But the crawlers are hitting an unknown brick wall. They think something
should be there and it's not, or the expected size of the file is different,
or...some such rot, and are being stopped dead in their tracks and
reporting that there is nothing there to be seen.

Eventually, your pages will be deindexed. Not a problem if you can
fix this quickly.

If you can change hosts with little or no extra cost, that might be your
last resort if your current host cannot help you.

It's not that some pages are not now being indexed. The whole
site cannot be crawled. The two above replies are just nonsense.

Paul

[ 1 ] Thanks
2 replies

Signature

If you were disappointed in your results today, lower your standards tomorrow.

{{ DiscussionBoard.errors[9819587].message }}

ClaudeMonet 9 years ago

Thank you. I think you are right.

Yesterday, I submitted an URL using the fetch as Google tool:

I had to submit the URL 3 times before it was found!!

Look at the screen:

New Hosting:

What about this offer? I don't want to spend too much money:

https://www.liquidweb.com/configure/...e-hosting-plan
- Thanks
- 1 reply
{{ DiscussionBoard.errors[9820643].message }}
- paulgl 9 years ago
  
  You may have lucked out and the problem got fixed. A good host
  will always be tracking things in the background.
  
  However, if it took that many times, it could be spotty, and rear
  its ugly head again. That also is detrimental, as eventually the crawlers
  give up for good.
  
  As of yesterday, looks like the site was crawled.
  
  Plugged it into the FB debugger, and it also is doing fine.
  
  Just keep checking.
  
  Paul
  
  [ 1 ] Thanks
  
  1 reply
  
  Signature
  
  If you were disappointed in your results today, lower your standards tomorrow.
  
  {{ DiscussionBoard.errors[9820739].message }}
  
  ClaudeMonet 9 years ago
  
  Thank you!
  
  I think I would feel better changing the host! Godaddy has made more than just one problem in the past.
  
  What about this liquidweb product? Is this an option for me? What do you think?
  
  Thanks
  
  {{ DiscussionBoard.errors[9820750].message }}

ClaudeMonet

9 years ago

Originally Posted by paulgl

The fact that a debugger, like the free on at FB shows a

"Document returned no data"

Shows that it is either a server problem or an authentication problem,
which could also be related to a server problem. That is, your
host.

You can access the pages, as well as probably 99% of the viewers.
Browsers are very, very forgiving in 2015.

But the crawlers are hitting an unknown brick wall. They think something
should be there and it's not, or the expected size of the file is different,
or...some such rot, and are being stopped dead in their tracks and
reporting that there is nothing there to be seen.

Eventually, your pages will be deindexed. Not a problem if you can
fix this quickly.

If you can change hosts with little or no extra cost, that might be your
last resort if your current host cannot help you.

It's not that some pages are not now being indexed. The whole
site cannot be crawled. The two above replies are just nonsense.

Paul

Proud to say that I'm a liquidweb customer since yesterday. I hope there are no server problems in the future. It looks good for the very first moment.

Thanks

{{ DiscussionBoard.errors[9825466].message }}

ClaudeMonet

9 years ago

Update - this morning:

Webmaster tool says:

149 URLs submitted
84 URLs indexed

I just find 70, when I run site:domain.com

Maybe it's because of the different server locations. I will let you know! I pray!

Thanks

{{ DiscussionBoard.errors[9821825].message }}

ajmalkhan

9 years ago

acording me it is robot.txt problem because i had never used a tag called allow in robot.txt it should be like this
useragent:*
disallow:
i have rad some hosting companies didn't updated the allow tag which is new in robot.txt

Thanks

{{ DiscussionBoard.errors[9824056].message }}

Hudson White

9 years ago

I agreed with ajmalkhan. Try using the following robots.txt file:

useragent:*
disallow:

N also do one more thing. Go in webmasters and click FETCH & Render this will help you.

Thanks
1 reply

{{ DiscussionBoard.errors[9824305].message }}

ClaudeMonet 9 years ago

I changed the robots.txt, let's see what happens.

But I guess, it's also because of my link profile. For now I just have content pages without links. And I used the fetch tool without getting the other pages indexed.
- Thanks
{{ DiscussionBoard.errors[9825464].message }}

yukon

Banned 9 years ago

I doubt the original problem was a host issue considering the pages were loading in a browser. If we can see a page, so can Google.

Thanks

{{ DiscussionBoard.errors[9825533].message }}

Ashwaniseo

9 years ago

[DELETED]

{{ DiscussionBoard.errors[9826747].message }}

ClaudeMonet

9 years ago

Originally Posted by Ashwaniseo

The primary thing you should to observe is your Google Webmaster Tools dashboard. On the off chance that there are issues, If there are issues, the dashboard will show you the error messages.

I'm in despair!

Look no errors and instead of adding more sites to the index, Google reduces the amount of indexed pages.

Furthermore it says 84 pages indexed but it's just between 65 and 69 depending on my request using site:www.mydomain.com or site:mydomain.com

Thanks

{{ DiscussionBoard.errors[9827478].message }}

ClaudeMonet

9 years ago

I just wanted to say THANK YOU VERY MUCH to all here. Indexing my pages makes progress. The number of indexed pages increased from 70 to 95. I think that's a good sign!

Thanks

{{ DiscussionBoard.errors[9832797].message }}

Problems with indexing my inside pages - Google

Trending Topics

A healthy fascination for Hollywood...

How did you turn it all around?

Any thoughts on golf's latest phenom...Scottie Scheffler??

best high quality crypto traffic.

What's the Best IM-Related Skill to Learn Today That Can Help in the Future?