need help abt robot.txt

5 replies
  • SEO
  • |
what it mean can anybody plz tell...
User-agent: *
allow: /wp-admin/

Sitemap: http://www.mydomain.com/sitemap.xml
#abt #robottxt
  • Profile picture of the author ajmalkhan
    It means that it allows all bots crawl your site and allowing admine also excess the site i think this what it is saying
    {{ DiscussionBoard.errors[9792593].message }}
  • Profile picture of the author BenJackson
    That is telling all bots crawling your site (including googlebot) to crawl all pages of your site. The second line is unnecessary and actually incorrect.

    Here's a decent configuration for a robots.txt file on a WordPress site:

    User-agent: *
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /xmlrpc.php
    Disallow: /wp-
    Allow: /wp-content/uploads/

    You can replace your existing robots.txt (except for the sitemap!) with the above.
    {{ DiscussionBoard.errors[9793195].message }}
  • Profile picture of the author SEO Power
    Originally Posted by jsam757 View Post

    what it mean can anybody plz tell...
    User-agent: *
    allow: /wp-admin/

    Sitemap: http://www.mydomain.com/sitemap.xml
    It should be disallow, not allow. If you upload it that way, Google will crawl your admin login page and the chances of your site getting hacked will increase.
    {{ DiscussionBoard.errors[9793266].message }}
    • Profile picture of the author BenJackson
      Originally Posted by SEO Power View Post

      If you upload it that way, Google will crawl your admin login page and the chances of your site getting hacked will increase.
      No, that's simply not true.
      {{ DiscussionBoard.errors[9793631].message }}
  • Profile picture of the author yukon
    Banned
    Lol, why would you even want a login page indexed?
    Signature
    Hi
    {{ DiscussionBoard.errors[9793728].message }}

Trending Topics