Robots.txt issue

15 years ago

Originally Posted by BigG95

I really need some help here. I have a wordpress site, which has been ranked # 1 in google for the last year or so.

Just yesterday it dropped to nowhere to be found.

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

My other sites, all wordpress all hosted with hostgator, do not have issues.

Help???

check your hosting provider,if your hosting provider has website builder sometime the problem could be in the Meta Data section , see i used joomla conected with M2host to me m2host is the best host on the net,just because you pay 12dollars a year for the hosting ,an you can shows different website builder like , joomla ,mambo,an much more but anyway check your hosting

Thanks

{{ DiscussionBoard.errors[3687060].message }}

RemingtonSteele

15 years ago

Originally Posted by BigG95

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

If Google is saying that the file is unreachable, then you likely don't have one, or it's in the wrong place. The robots.txt file is supposed to go in the root of your site (e.g., http://www.yoursite.com/robots.txt). What happens when you go to the URL above (edited for your own domain, of course)? If you get a 404 error, then your robots.txt file is missing. To add it, just create one and upload it. Simple as that.

There are tons of robots.txt tutorials online that should help you figure out what to put in your file.

Thanks
1 reply

{{ DiscussionBoard.errors[3687263].message }}

BigG95

15 years ago

Ok, checked everything I can. Site is still indexed in google, just nowhere to be found. robots.txt file looks ok.

if i type in mysite.com/robots.txt it shows
User-agent: * Disallow: Sitemap: http://mysite.com/sitemap.xml.gz

Still google bot can't find the robots.txt file?

Am I missing something?

Thanks
2 replies

Signature

http://SmartFoxMarketingGroup.com

{{ DiscussionBoard.errors[3687488].message }}

BigG95

15 years ago

ok, I resubmitted the site map to google webmaster account, checked and backchecked everything.

Google bot still can't find the robots.txt.

Any ideas?

Thanks
1 reply

Signature

http://SmartFoxMarketingGroup.com

{{ DiscussionBoard.errors[3688353].message }}

RemingtonSteele

15 years ago

It could be that your web host's firewall is blocking IPs that make "too many" requests within a short period of time. Maybe you should contact your host and inquire about this issue.

More info: http://www.homewithandrew.com/index.php/debugging-the-network-unreachable-robots-txt-unreachable-error/

[ 1 ] Thanks
2 replies

{{ DiscussionBoard.errors[3693629].message }}

paulgl

15 years ago

robots.txt has nothing to do with SERPs.
If it's not found, no big deal. Googlebot will try
and find it first. I have never had one. No need.
It will show a not found error. But again, unless
it just creeps you out to have a not found error,
it's no big deal.

Well, I guess it may if you had some stuff that you
did not want indexed. But then that would be a different
matter.

Paul

Thanks
2 replies

Signature

If you were disappointed in your results today, lower your standards tomorrow.

{{ DiscussionBoard.errors[3693897].message }}

BigG95 15 years ago

@ Paul. Are you sure it has nothing to do with my site dropping in the Serps. Since it started dropping shortly after google bot could not access my robots.txt anymore
- Thanks
Signature

http://SmartFoxMarketingGroup.com
{{ DiscussionBoard.errors[3714324].message }}

guitarjosh

15 years ago

Originally Posted by paulgl

robots.txt has nothing to do with SERPs.
If it's not found, no big deal. Googlebot will try
and find it first. I have never had one. No need.
It will show a not found error. But again, unless
it just creeps you out to have a not found error,
it's no big deal.

Well, I guess it may if you had some stuff that you
did not want indexed. But then that would be a different
matter.

Paul

Kind of right, but in this case.. completely wrong. If you have a robots.txt file and it returns an "ambiguous" error as it's seemingly doing in this case, this is the problem. If Google sees there's a robots.txt but can't read it, they wisely assume your content is blocked (it wouldn't be a good practice to say "we can't read it, so let's index the whole thing").

If you can't solve the issue with the robots.txt file, you're better off to delete it. Then... it becomes "no big deal". Otherwise, your site will simply no longer be crawled... and that's a pretty big deal.

Source: robots.txt unreachable - Webmaster Central Help

Thanks

{{ DiscussionBoard.errors[3715082].message }}

BigG95 15 years ago

@ Remington Steel, thank you, I contacted my Host. I could not find any errors on my site.
- Thanks
Signature

http://SmartFoxMarketingGroup.com
{{ DiscussionBoard.errors[3714317].message }}

timpears 15 years ago

Originally Posted by BigG95

if i type in mysite.com/robots.txt it shows
User-agent: * Disallow: Sitemap: http://mysite.com/sitemap.xml.gz

Why would you want to disallow the site map? You want Google to read your site map.
- Thanks
- 1 reply
Signature

Tim Pears
{{ DiscussionBoard.errors[3715180].message }}
- guitarjosh 15 years ago
  
  Originally Posted by timpears
  
  Why would you want to disallow the site map? You want Google to read your site map.
  
  He's not disallowing it. His robots file is correct. It reads (as it should):
  
  User-agent: *
  Disallow:
  Sitemap: http://mysite.com/sitemap.xml.gz
  
  Thanks
  
  1 reply
  
  {{ DiscussionBoard.errors[3715462].message }}
  
  timpears 15 years ago
  
  Originally Posted by guitarjosh
  
  He's not disallowing it. His robots file is correct. It reads (as it should):
  
  User-agent: *
  Disallow:
  Sitemap: http://mysite.com/sitemap.xml.gz
  
  I guess I misread it. I don't really understand robots.text that well.
  
  Thanks
  
  Signature
  
  Tim Pears
  
  {{ DiscussionBoard.errors[3715646].message }}

yukon

Banned 15 years ago

When you have a GWT robots.txt error, don't use the resubmit option.

1) Double check your url is good by searching the exact robots.txt url

2) If the robots.txt url is all good, delete the old robots.txt link inside GWT

3) Paste the good robots.txt url into GWT again (submit (not resubmit)), refresh your browser a couple of times.

I'm not sure why this happens but I've found that the GWT resubmit option doesn't work very often. When I've just deleted & started over (submit) it works just fine.

Thanks

{{ DiscussionBoard.errors[3694699].message }}

syahbiz

15 years ago

i dont understand how to use this

Thanks

Signature

cheers!!
Auto Insurance Tips

{{ DiscussionBoard.errors[3695424].message }}

ankitsharma

15 years ago

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com would not apply to a.example.com.

Thanks
1 reply

{{ DiscussionBoard.errors[3704798].message }}

BigG95 15 years ago

@ yukon, I have tried this, no luck, but thanks
- Thanks
Signature

http://SmartFoxMarketingGroup.com
{{ DiscussionBoard.errors[3714385].message }}

rvitgroup

15 years ago

hello everyone,

I know nothing about how to recover such an position but after reading the quires and their response now i understand some basic of it. So well job done by all.

Thanks

{{ DiscussionBoard.errors[3715331].message }}

ianspencer

15 years ago

first of all you need to check that robots file is working or not just type domainname/robots.txt if it is working then there is no problem and if not working then check your file where it is this file will be on root.

Thanks

Signature

Childrens Bedding

{{ DiscussionBoard.errors[3715587].message }}

rising_sun

Banned 15 years ago

Originally Posted by BigG95

I really need some help here. I have a wordpress site, which has been ranked # 1 in google for the last year or so.

Just yesterday it dropped to nowhere to be found.

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

My other sites, all wordpress all hosted with hostgator, do not have issues.

Help???

Of course me too say to you the same things check the index from your hosting provider. I think they will know what is happening inside there. Please knock them quickly. Hopefully you will get index very first.

Thanks
1 reply

{{ DiscussionBoard.errors[3715875].message }}

BigG95

15 years ago

Ok, I checked with Hostgator, they were blocking google bot, I meanwhile entered all of my sites in Webmaster account twice.

Once as mysite.com and once as www.mysite.com. According to google, they treat those 2 as different websites.

I set the www. as preferred, also according to google, otherwise I might get punished for duplicate content.

Here comes the fun part now:

For some of my sites, google can find the robots.txt now for both the www.mysite.com and the mysite.com.

For other sites they can only find the www.mysite.com/robots.txt but not the other one.

I do however still have one site, where google bot can't find the robots.txt on neither.

Does not make sense at all, I know.

@guitarjosh - I would like to try to remove the robots.txt for the one site, I mentioned, but I have wordpress installed and they somehow automatically add it.
Searched through Wordpress Forums, but no luck. I need to know, how to remove the robots.txt from my wordpress site.

Thanks
1 reply

Signature

http://SmartFoxMarketingGroup.com

{{ DiscussionBoard.errors[3717151].message }}

guitarjosh

15 years ago

Originally Posted by BigG95

Ok, I checked with Hostgator, they were blocking google bot, I meanwhile entered all of my sites in Webmaster account twice.

Once as mysite.com and once as www.mysite.com. According to google, they treat those 2 as different websites.

I set the www. as preferred, also according to google, otherwise I might get punished for duplicate content.

Here comes the fun part now:

For some of my sites, google can find the robots.txt now for both the www.mysite.com and the mysite.com.

For other sites they can only find the www.mysite.com/robots.txt but not the other one.

I do however still have one site, where google bot can't find the robots.txt on neither.

Does not make sense at all, I know.

@guitarjosh - I would like to try to remove the robots.txt for the one site, I mentioned, but I have wordpress installed and they somehow automatically add it.
Searched through Wordpress Forums, but no luck. I need to know, how to remove the robots.txt from my wordpress site.

Wordpress's "virtual" robots.txt file is a pain in the arse. Just manually create a robots.txt file and put it in the root of your site. That should override the virtual.

How did you set the www as preferred in HostGator? I haven't checked in to seeing how mine's set but thought maybe I should.

Thanks

{{ DiscussionBoard.errors[3760420].message }}

Robots.txt issue

Trending Topics

Clients Ghosting: Advice please!

The Winter Olympics....

Best SEO strategy for wedding venue websites

Help! My domain has been hijacked by a spammer.

How important is local SEO for freelancers in 2026?