5 {{ upvoteCount | shortNum }}
5 {{ upvoteCount | shortNum }}

Techie one about robots.txt

by phil.wheatley
4 replies
Hi Folks

1 of my sites won't get indexed, google tools says it's due to something ni the robots.txt blockling Google from crwaling it. It says to edit it or remove it. I've looked via ftp but there are no files of that sort. Can I only see this via google tools then? But that means having to viefty sites and all that.

2nd thing is as I don't want to block anhything from Google, is it ok just to delete the robots.txt file entirely...on the assumption I can even find the bl**dy thing? As a side note, I got that iframe virus a while ago and wondering if it's to do with that? I thought I had cleaned it all but who knows.

Thanks,
Phil
#robotstxt #techie
  • Profile picture of the author Tyrus Antas
    Unless you're trying to keep some content away from Google's claws You are safe to delete it. The file should be place inside your root directory. If you have directories as subdomains and Google isn't being able to access your subdomains, then the robots.txt should be inside the corresponding subdirectory.

    Tyrus
    {{ DiscussionBoard.errors[756898].message }}
  • Profile picture of the author Devon Brown
    Are you double-checking the root folder of your FTP and not just the root folder of that one website?
    Signature
    Click Here To Make $2.63 - $5.93 Per Click As A JV/Affiliate!!
    Follow Me On Twitter, Facebook [/B],Visit My Blog
    {{ DiscussionBoard.errors[756952].message }}
  • Profile picture of the author TheRichJerksNet
    You can delete if you wish to but it is good to have a robots.txt file for several reasons...

    1. Block spiders from certain folders
    2. Direct spiders to your site map

    See my post here:
    http://www.warriorforum.com/main-int...tml#post755573

    A robots.txt file that has the following will block spiders
    User-agent: *
    Disallow: *

    Now mind you not all spiders listen to robots.txt files.. But adding your sitemap (explained in the link above) to the robots.txt file will help yahoo and google index your pages better.

    James
    {{ DiscussionBoard.errors[757056].message }}
  • Profile picture of the author phil.wheatley
    Thanks for your replies guys. James, I'm just going through your instructions now.

    Cheers,
    Phil
    Signature



    It's still not working for you??? Need direction?...
    ---->>>> BrainDirection.com <<<<----
    {{ DiscussionBoard.errors[757495].message }}

Trending Topics