Robots.txt issue

[BigG95]

I really need some help here. I have a wordpress site, which has been ranked # 1 in google for the last year or so.

Just yesterday it dropped to nowhere to be found.

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

My other sites, all wordpress all hosted with hostgator, do not have issues.

Help???

[tom johnson]

Originally Posted by BigG95

I really need some help here. I have a wordpress site, which has been ranked # 1 in google for the last year or so.

Just yesterday it dropped to nowhere to be found.

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

My other sites, all wordpress all hosted with hostgator, do not have issues.

Help???

check your hosting provider,if your hosting provider has website builder sometime the problem could be in the Meta Data section , see i used joomla conected with M2host to me m2host is the best host on the net,just because you pay 12dollars a year for the hosting ,an you can shows different website builder like , joomla ,mambo,an much more but anyway check your hosting

[RemingtonSteele]

Originally Posted by BigG95

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

If Google is saying that the file is unreachable, then you likely don't have one, or it's in the wrong place. The robots.txt file is supposed to go in the root of your site (e.g., http://www.yoursite.com/robots.txt). What happens when you go to the URL above (edited for your own domain, of course)? If you get a 404 error, then your robots.txt file is missing. To add it, just create one and upload it. Simple as that.

There are tons of robots.txt tutorials online that should help you figure out what to put in your file.

[BigG95]

Ok, checked everything I can. Site is still indexed in google, just nowhere to be found. robots.txt file looks ok.

if i type in mysite.com/robots.txt it shows
User-agent: * Disallow: Sitemap: http://mysite.com/sitemap.xml.gz

Still google bot can't find the robots.txt file?

Am I missing something?

[BigG95]

ok, I resubmitted the site map to google webmaster account, checked and backchecked everything.

Google bot still can't find the robots.txt.

Any ideas?

[RemingtonSteele]

It could be that your web host's firewall is blocking IPs that make "too many" requests within a short period of time. Maybe you should contact your host and inquire about this issue.

More info: http://www.homewithandrew.com/index.php/debugging-the-network-unreachable-robots-txt-unreachable-error/

[paulgl]

robots.txt has nothing to do with SERPs.
If it's not found, no big deal. Googlebot will try
and find it first. I have never had one. No need.
It will show a not found error. But again, unless
it just creeps you out to have a not found error,
it's no big deal.

Well, I guess it may if you had some stuff that you
did not want indexed. But then that would be a different
matter.

Paul

[yukon]

When you have a GWT robots.txt error, don't use the resubmit option.

1) Double check your url is good by searching the exact robots.txt url

2) If the robots.txt url is all good, delete the old robots.txt link inside GWT

3) Paste the good robots.txt url into GWT again (submit (not resubmit)), refresh your browser a couple of times.

I'm not sure why this happens but I've found that the GWT resubmit option doesn't work very often. When I've just deleted & started over (submit) it works just fine.

[syahbiz]

i dont understand how to use this

[ankitsharma]

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com would not apply to a.example.com.

[BigG95]

@ Remington Steel, thank you, I contacted my Host. I could not find any errors on my site.

[BigG95]

@ Paul. Are you sure it has nothing to do with my site dropping in the Serps. Since it started dropping shortly after google bot could not access my robots.txt anymore

[BigG95]

@ yukon, I have tried this, no luck, but thanks

[guitarjosh]

Originally Posted by paulgl

robots.txt has nothing to do with SERPs.
If it's not found, no big deal. Googlebot will try
and find it first. I have never had one. No need.
It will show a not found error. But again, unless
it just creeps you out to have a not found error,
it's no big deal.

Well, I guess it may if you had some stuff that you
did not want indexed. But then that would be a different
matter.

Paul

Kind of right, but in this case.. completely wrong. If you have a robots.txt file and it returns an "ambiguous" error as it's seemingly doing in this case, this is the problem. If Google sees there's a robots.txt but can't read it, they wisely assume your content is blocked (it wouldn't be a good practice to say "we can't read it, so let's index the whole thing").

If you can't solve the issue with the robots.txt file, you're better off to delete it. Then... it becomes "no big deal". Otherwise, your site will simply no longer be crawled... and that's a pretty big deal.

Source: robots.txt unreachable - Webmaster Central Help

[timpears]

Originally Posted by BigG95

if i type in mysite.com/robots.txt it shows
User-agent: * Disallow: Sitemap: http://mysite.com/sitemap.xml.gz

Why would you want to disallow the site map? You want Google to read your site map.

[rvitgroup]

hello everyone,

I know nothing about how to recover such an position but after reading the quires and their response now i understand some basic of it. So well job done by all.

[guitarjosh]

Originally Posted by timpears

Why would you want to disallow the site map? You want Google to read your site map.

He's not disallowing it. His robots file is correct. It reads (as it should):

User-agent: *
Disallow:
Sitemap: http://mysite.com/sitemap.xml.gz

[ianspencer]

first of all you need to check that robots file is working or not just type domainname/robots.txt if it is working then there is no problem and if not working then check your file where it is this file will be on root.

[timpears]

Originally Posted by guitarjosh

He's not disallowing it. His robots file is correct. It reads (as it should):

User-agent: *
Disallow:
Sitemap: http://mysite.com/sitemap.xml.gz

I guess I misread it. I don't really understand robots.text that well.

[rising_sun]

Originally Posted by BigG95

I really need some help here. I have a wordpress site, which has been ranked # 1 in google for the last year or so.

Just yesterday it dropped to nowhere to be found.

I checked google webmaster tools and its says robots.txt unreachable.

I have no clue, how to fix this.

My other sites, all wordpress all hosted with hostgator, do not have issues.

Help???

Of course me too say to you the same things check the index from your hosting provider. I think they will know what is happening inside there. Please knock them quickly. Hopefully you will get index very first.

[BigG95]

Ok, I checked with Hostgator, they were blocking google bot, I meanwhile entered all of my sites in Webmaster account twice.

Once as mysite.com and once as www.mysite.com. According to google, they treat those 2 as different websites.

I set the www. as preferred, also according to google, otherwise I might get punished for duplicate content.

Here comes the fun part now:

For some of my sites, google can find the robots.txt now for both the www.mysite.com and the mysite.com.

For other sites they can only find the www.mysite.com/robots.txt but not the other one.

I do however still have one site, where google bot can't find the robots.txt on neither.

Does not make sense at all, I know.

@guitarjosh - I would like to try to remove the robots.txt for the one site, I mentioned, but I have wordpress installed and they somehow automatically add it.
Searched through Wordpress Forums, but no luck. I need to know, how to remove the robots.txt from my wordpress site.

[guitarjosh]

Originally Posted by BigG95

Ok, I checked with Hostgator, they were blocking google bot, I meanwhile entered all of my sites in Webmaster account twice.

Once as mysite.com and once as www.mysite.com. According to google, they treat those 2 as different websites.

I set the www. as preferred, also according to google, otherwise I might get punished for duplicate content.

Here comes the fun part now:

For some of my sites, google can find the robots.txt now for both the www.mysite.com and the mysite.com.

For other sites they can only find the www.mysite.com/robots.txt but not the other one.

I do however still have one site, where google bot can't find the robots.txt on neither.

Does not make sense at all, I know.

@guitarjosh - I would like to try to remove the robots.txt for the one site, I mentioned, but I have wordpress installed and they somehow automatically add it.
Searched through Wordpress Forums, but no luck. I need to know, how to remove the robots.txt from my wordpress site.

Wordpress's "virtual" robots.txt file is a pain in the arse. Just manually create a robots.txt file and put it in the root of your site. That should override the virtual.

How did you set the www as preferred in HostGator? I haven't checked in to seeing how mine's set but thought maybe I should.