Allowing online seo analytics tools to crawl my website?

4 replies
  • SEO
  • |
I have disallowed all the bots to crawl my website because it is under construction.What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.The website has some important company data they don't want anyone to access, that's why i can't allow all.

ErrorThis URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.
#allowing #analytics #crawl #online #seo #tools #website
  • Profile picture of the author counter2015
    Add no index to all the pages and allow the bots that should resolve the issue
    Signature
    {{ DiscussionBoard.errors[10358163].message }}
  • Profile picture of the author yukon
    Banned
    Originally Posted by janiman View Post

    I have disallowed all the bots to crawl my website because it is under construction.What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.The website has some important company data they don't want anyone to access, that's why i can't allow all.

    ErrorThis URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.

    The company data isn't too important If it's a on a public server.
    {{ DiscussionBoard.errors[10358178].message }}
    • Profile picture of the author paulgl
      Why would you care about seo if you want the stuff hidden?

      You want it crawled, but not really.

      Does not matter what you do with robots.txt

      Google can and does index content that robots.txt blocks.

      Why is anything that is company data that you want hidden even in the public?

      If you don't know how to make company data secret, that needs to be secret, man your company needs a new man.

      Paul
      Signature

      If you were disappointed in your results today, lower your standards tomorrow.

      {{ DiscussionBoard.errors[10358322].message }}
  • Profile picture of the author danparks
    Originally Posted by janiman View Post

    The website has some important company data they don't want anyone to access, that's why i can't allow all.
    Paul's reply pretty much says it all. But I'll put some more emphasis on it. You need to learn how to make info that is supposed to be private, private. Just hiding it from Google doesn't make it private. That only means it won't show up in a Google search. Anyone who can figure out the page URL can still simply type that URL into a browser and see the page. You really, really, really need to learn about making pages private. For the sake of your company, and the sake of your job.
    {{ DiscussionBoard.errors[10358508].message }}

Trending Topics