Google-bot Can't Access my website?

Banned 10 years ago

Originally Posted by aadi14

There is a travel website having about 13,000 destination pages like from A to B and B to A, C to B and C to A and continues. But, the sitemap can't index all the website pages and its also showing that Google-bot can't access your site. What can I do to resolve the problem? Not even half of the pages are indexed. Also, for such a big website Should I add different sitemaps ? I already added 10. As in a sitemap only 8,000 are allowed. How can I fix this error ? I am getting good traffic though but I am scared my website will get banned ? Is that so? The website has only 6 pages and the journey will only show up once the destinations are entered.

Well, I think there is some problem with the Robots.txt file try checking the robots.txt file for errors once the robots.txt file is fixed try creating a sitemap again and submit over to the webmaster tool, wait a while to see if Google bot can now access your site ??

Thanks
1 reply

{{ DiscussionBoard.errors[9382242].message }}

aadi14

10 years ago

Originally Posted by promo87

Well, I think there is some problem with the Robots.txt file try checking the robots.txt file for errors once the robots.txt file is fixed try creating a sitemap again and submit over to the webmaster tool, wait a while to see if Google bot can now access your site ??

Checked the robots.txt its like this :
#robots.txt for <http://www.abc.com>
User-agent:*
Disallow: /backoffice/

Sitemap: <http://www.abc.com/sitemap1.xml>
Sitemap: <http://www.abc.com/sitemap2.xml>

Just like this there are a total of 228 entries in sitemap.(excluding <> signs)

is this the right way ? as there are more than 13,000 journey the website provides so have to submit so many sitemaps.

Thanks

Signature

Ecommerce Website Designing Company | Bespoke Website & Application Development Company

{{ DiscussionBoard.errors[9382274].message }}

sanusense

10 years ago

Checking up Robots.txt page will help you and use Internal Linking, bots can't crawl all your pages if there is something programming mistake, so just take precautions and do Internal linking.

A to B, B-C, C-A.
So, if search engine bot will come to index your page A, it will see another link B. CRAWLING DONE.

Then again when it will reach Link B it will again see Link C, DONE

Problem Solved!!!

Thanks

{{ DiscussionBoard.errors[9382249].message }}

Scott016

10 years ago

It has to be the robot.txt file. Make sure that file is properly made . After you make the robot.txt file , resubmit sitemap and also use " Fetch as Google " option in webmasters to fetch your website and certain sub domains. I hope that works for you.

Thanks

{{ DiscussionBoard.errors[9382250].message }}

hieuvu899

10 years ago

problems of your are in the file robots.txt, you use edit tập tin of this to google can read information from sitemap

Thanks

{{ DiscussionBoard.errors[9382285].message }}

alexjames212

10 years ago

when did you submit the sitemap, date?

Thanks
2 replies

Signature

French Bulldog Care

{{ DiscussionBoard.errors[9382308].message }}

versasia 10 years ago

Try this format. May be it will helpful to you. User-Agent: * Disallow: /backoffice/ Sitemap: http://www.abc.com/sitemap.xml
- Thanks
{{ DiscussionBoard.errors[9382667].message }}
aadi14 10 years ago

Originally Posted by alexjames212

when did you submit the sitemap, date?

Its been so long. I have submitted it last year in February.
- Thanks
- 1 reply
Signature
Ecommerce Website Designing Company | Bespoke Website & Application Development Company
{{ DiscussionBoard.errors[9392934].message }}
- seopratyush 10 years ago
  
  You can test your robots.txt file in the webmasters with the help of rorbots tester and see if there is any error. It also suggest you the type of error. And then you can be able to upload the correct version of file.
  
  [ 1 ] Thanks
  
  {{ DiscussionBoard.errors[9394624].message }}

SEO Power

10 years ago

Extensive internal linking and authority are very important for deeper indexing. Internal linking helps Googlebot find new pages faster and authority (determined by the number and PR of backlinks) help with deeper crawling and frequent crawling.

Make sure the pages that haven't been indexed yet aren't blocked by your robots.txt file, then build some backlinks to them and give Googlebot some time to visit and crawl them.

Another potential culprit could be the reliability of your host. If your website is frequently down whenever Google tries to index your pages, most of your pages won't be indexed.

Thanks

Signature

ONE HOUR INDEXING - STOP WASTING MONEY ON LINKS GOOGLE NEVER FINDS

{{ DiscussionBoard.errors[9382938].message }}

mkgg

10 years ago

Its not his robots.txt, jeez do you guys even read ?.

Fetch as googlebot in WMT and notice the error code that you receive, is it 404/200/403 which one ? What response does googlebot receive. If googlebot is fetching fine then its your server being crappy and probably because of downtimes.

You can also check your robots.txt in WMT, google will tell you if its good or not. But robots.txt is not the issue. The only thing your robots.txt is blocking right now is backoffice folder which i suppose is what you want right ?.

Also if robots.txt was the problem, google your site and it should show a message that google can't show webpage because it is not allowed access

Thanks
1 reply

{{ DiscussionBoard.errors[9395697].message }}

aadi14

10 years ago

Originally Posted by mkgg

Its not his robots.txt, jeez do you guys even read ?.

Fetch as googlebot in WMT and notice the error code that you receive, is it 404/200/403 which one ? What response does googlebot receive. If googlebot is fetching fine then its your server being crappy and probably because of downtimes.

You can also check your robots.txt in WMT, google will tell you if its good or not. But robots.txt is not the issue. The only thing your robots.txt is blocking right now is backoffice folder which i suppose is what you want right ?.

Also if robots.txt was the problem, google your site and it should show a message that google can't show webpage because it is not allowed access

Will try the fetch instruction in WMT first. and I want to block the Backoffice though as its the admin panel link.