robots.txt

by 2 replies
2
Hi just a quick question.... this is what google says my robots.txt file is


User-agent: *
Disallow:


I thought this meant to allow googlebot right? because gwt is telling me 16 of my pages are restricted... does this robot.txt indeed allow or not google to crawl??? Ive done a thorough forum search but info is confusing thanks
#search engine optimization #robotstxt
  • That doesn't block anything the way it's written, another words, nothing is disallowed. It's the same as not having a robots.txt file at all.

    This would tell all the robots not to visit any pages on the site. Bad for SEO lol

    User-agent: *
    Disallow: /
    • [1] reply
    • thanks 2stroke guess ill jsut wait for google to recrawl site...
Join the discussion

Next Topics on Trending Feed