Need help with my robots.txt file please...

Si_P · 10-10-2009, 04:03 AM

Hi,

Like many of my other money sites, they are hosted with Hostagator and in the form of Wordpress blog. Never have I encountered this problem before as i have never had to set up a robots.txt file manually.

For some reason when I try to submit my sitemap to google, i get a crawl error that my robots.txt file has the txt:

User-agent: *
Disallow: /

which because of the '/' after 'Disallow:', my complete site cannot be crawled.

I never set this up to be like that and i havent a clue how to change it. I decided to manually create a new robots.txt file with:

User-agent: *
Disallow:

so that my site should be crawled. I uploaded it to the '/public_html/mydomain dot com' directory about 10 hours ago, but it now lists all the page addresses which cannot be accessed and shows the following after testing the file in my webmaster tools section.

Url Googlebot Googlebot-Mobile mydomain dot com/ Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions
Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions

How on earth do i fix this as i need my site indexed asap by the Googlebot?

Thanks,
Si

Bruce Hearder · 10-10-2009, 04:38 AM

You can use both formats :

User-agent: *
Disallow:

or

User-agent: *
Allow: /

Sometimes it takes BigG a whiole to come back again.
I would make sure you log into Google Webmasters, and go the domain in question.
Click on "Site Configuration" , then "Crawler Access".

At the bottom, of that page is a "Test" button.

Click that to make sure that Google can read your robots.txt file.

Also, a catch for "young players" is that in the latest version of WP (2.8.4) its privacy default is set so that Search Engines can't read your site.

Go to Settings->Privacy and check which option is checked.

The default is :

"I would like to block search engines, but allow normal visitors"

It should be changed to "I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers"

Then press the "Save Changes Button"

I hope this helps

Bruce

Si_P · 10-11-2009, 05:57 AM

Hi Bruce,

Thanks for the info. I had the new version of Wordpress on the site so had to change the privacy settings.

It is allowed now.

10-10-2009, 04:03 AM	#1
Si_P I'm trying... Join Date: Jul 2009 Location: UK Posts: 187 Thanks: 1 Thanked 5 Times in 4 Posts	Need help with my robots.txt file please... Hi, Like many of my other money sites, they are hosted with Hostagator and in the form of Wordpress blog. Never have I encountered this problem before as i have never had to set up a robots.txt file manually. For some reason when I try to submit my sitemap to google, i get a crawl error that my robots.txt file has the txt: User-agent: * Disallow: / which because of the '/' after 'Disallow:', my complete site cannot be crawled. I never set this up to be like that and i havent a clue how to change it. I decided to manually create a new robots.txt file with: User-agent: * Disallow: so that my site should be crawled. I uploaded it to the '/public_html/mydomain dot com' directory about 10 hours ago, but it now lists all the page addresses which cannot be accessed and shows the following after testing the file in my webmaster tools section. Url Googlebot Googlebot-Mobile mydomain dot com/ Blocked by line 2: Disallow: / Detected as a directory; specific files may have different restrictions Blocked by line 2: Disallow: / Detected as a directory; specific files may have different restrictions How on earth do i fix this as i need my site indexed asap by the Googlebot? Thanks, Si

10-10-2009, 04:38 AM	#2
Bruce Hearder Advanced Warrior War Room Member Join Date: May 2004 Location: Perth, Australia. Posts: 717 Thanks: 4 Thanked 182 Times in 138 Posts Social Networking Contact Info	Re: Need help with my robots.txt file please... You can use both formats : User-agent: * Disallow: or User-agent: * Allow: / Sometimes it takes BigG a whiole to come back again. I would make sure you log into Google Webmasters, and go the domain in question. Click on "Site Configuration" , then "Crawler Access". At the bottom, of that page is a "Test" button. Click that to make sure that Google can read your robots.txt file. Also, a catch for "young players" is that in the latest version of WP (2.8.4) its privacy default is set so that Search Engines can't read your site. Go to Settings->Privacy and check which option is checked. The default is : "I would like to block search engines, but allow normal visitors" It should be changed to "I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers" Then press the "Save Changes Button" I hope this helps Bruce
	----------------- Get Your Backlinks indexed quicker at BackLinks2RSS Create Full Text Feeds from Partial RSS Feeds at FeedExpander.com. See the WarriorForum post about it here

10-11-2009, 05:57 AM	#3
Si_P I'm trying... Join Date: Jul 2009 Location: UK Posts: 187 Thanks: 1 Thanked 5 Times in 4 Posts	Re: Need help with my robots.txt file please... Hi Bruce, Thanks for the info. I had the new version of Wordpress on the site so had to change the privacy settings. It is allowed now.