Go Back   WarriorForum - Internet Marketing Forums > Warrior Support Forums > Programming Talk
Register Blogs FAQ Social Groups CalendarHelp Desk

Reply
 
LinkBack Thread Tools
Old 10-10-2009, 04:03 AM   #1
I'm trying...
 
Join Date: Jul 2009
Location: UK
Posts: 187
Thanks: 1
Thanked 5 Times in 4 Posts
Default Need help with my robots.txt file please...

Hi,

Like many of my other money sites, they are hosted with Hostagator and in the form of Wordpress blog. Never have I encountered this problem before as i have never had to set up a robots.txt file manually.

For some reason when I try to submit my sitemap to google, i get a crawl error that my robots.txt file has the txt:

User-agent: *
Disallow: /

which because of the '/' after 'Disallow:', my complete site cannot be crawled.

I never set this up to be like that and i havent a clue how to change it. I decided to manually create a new robots.txt file with:

User-agent: *
Disallow:

so that my site should be crawled. I uploaded it to the '/public_html/mydomain dot com' directory about 10 hours ago, but it now lists all the page addresses which cannot be accessed and shows the following after testing the file in my webmaster tools section.

Url Googlebot Googlebot-Mobile mydomain dot com/ Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions
Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions

How on earth do i fix this as i need my site indexed asap by the Googlebot?

Thanks,
Si
Si_P is offline   Reply With Quote
Old 10-10-2009, 04:38 AM   #2
Advanced Warrior
War Room Member
 
Bruce Hearder's Avatar
 
Join Date: May 2004
Location: Perth, Australia.
Posts: 717
Thanks: 4
Thanked 182 Times in 138 Posts
Social Networking View Member's Twitter Profile  View Member's YouTube Profile
Contact Info
Send a message via Skype™ to Bruce Hearder
Default Re: Need help with my robots.txt file please...

You can use both formats :

User-agent: *
Disallow:

or

User-agent: *
Allow: /

Sometimes it takes BigG a whiole to come back again.
I would make sure you log into Google Webmasters, and go the domain in question.
Click on "Site Configuration" , then "Crawler Access".

At the bottom, of that page is a "Test" button.

Click that to make sure that Google can read your robots.txt file.

Also, a catch for "young players" is that in the latest version of WP (2.8.4) its privacy default is set so that Search Engines can't read your site.

Go to Settings->Privacy and check which option is checked.

The default is :

"I would like to block search engines, but allow normal visitors"

It should be changed to "I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers"

Then press the "Save Changes Button"

I hope this helps

Bruce

-----------------
Get Your Backlinks indexed quicker at BackLinks2RSS

Create Full Text Feeds from Partial RSS Feeds at FeedExpander.com. See the WarriorForum post about it here
Bruce Hearder is offline   Reply With Quote
Old 10-11-2009, 05:57 AM   #3
I'm trying...
 
Join Date: Jul 2009
Location: UK
Posts: 187
Thanks: 1
Thanked 5 Times in 4 Posts
Default Re: Need help with my robots.txt file please...

Hi Bruce,

Thanks for the info. I had the new version of Wordpress on the site so had to change the privacy settings.

It is allowed now.
Si_P is offline   Reply With Quote
Reply

  WarriorForum - Internet Marketing Forums > Warrior Support Forums > Programming Talk

Tags
file, robotstxt

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off



All times are GMT -6. The time now is 04:17 AM.