6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

Robots.txt?

by SladeK
5 replies
  • SEO
    • |
I have seen that you should set the robots.txt to /allow to ensure google crawls your site?

First of all, where on your site typically is the robots.txt file? (I am using wordpress) or do you have to create it?

Second, is this true?
#robotstxt
  • Profile picture of the author Radix
    Originally Posted by SladeK View Post

    I have seen that you should set the robots.txt to /allow to ensure google crawls your site?

    First of all, where on your site typically is the robots.txt file? (I am using wordpress) or do you have to create it?

    Second, is this true?

    9 viewers and not one answer? Shame on you people.

    WordPress › KB Robots.txt WordPress Plugins

    just follow the instructions...pretty straightforward.
    Signature
    Outside of a dog, a book is a man's best friend. Inside of a dog it's too dark to read.
    -Groucho Marx
    {{ DiscussionBoard.errors[1424475].message }}
    • Profile picture of the author SladeK
      Thanks a lot.
      One question remains, is setting it to only allow Google the typical way to go? Or should I set it to allow all bots?
      {{ DiscussionBoard.errors[1424572].message }}
      • Profile picture of the author timpears
        Originally Posted by SladeK View Post

        Thanks a lot.
        One question remains, is setting it to only allow Google the typical way to go? Or should I set it to allow all bots?
        I know that Google owns the world, but there is a little bit of the search traffic that they don't control. Do you want to ignore that part or would you want users of Yahoo and Bing etc. to see your site in their lists too?
        Signature

        Tim Pears

        {{ DiscussionBoard.errors[1424610].message }}
        • Profile picture of the author dburk
          Hi SladeK,

          Wordpress has a built in feature that allows you to turn bot access on and off. It's under Settings -> privacy. You don't need a plugin to block or allow access unless you want to selectively allow access to certain pages or to block certain robots.
          Signature

          * Get Results - Outsource Your PPC Management |
          *           Don Burk Advertising & Marketing - www.donburk.com

          {{ DiscussionBoard.errors[1424632].message }}
          • Profile picture of the author Radix
            Originally Posted by dburk View Post

            Hi SladeK,

            Wordpress has a built in feature that allows you to turn bot access on and off. It's under Settings -> privacy. You don't need a plugin to block or allow access unless you want to selectively allow access to certain pages or to block certain robots.

            This is true, but I recently watched a video clip (can't find it now and it's pissing me off because I know I have the link) made by a google dude stating that they record the robot file when they grab the sitemap during initial reviews of sites and without it they can delay indexing while they verify with several more passes that one does or doesn't exist. Ultimately shouldn't matter unless you actually do prohibit access to some areas, but it might delay your indexing according to some dude that I can't find his video now.
            Signature
            Outside of a dog, a book is a man's best friend. Inside of a dog it's too dark to read.
            -Groucho Marx
            {{ DiscussionBoard.errors[1425893].message }}

Trending Topics