need help abt robot.txt

by jsam757

Posted: 11 years ago 5 replies

what it mean can anybody plz tell...
User-agent: *
allow: /wp-admin/

Sitemap: http://www.mydomain.com/sitemap.xml

#abt #robottxt

ajmalkhan 11 years ago

It means that it allows all bots crawl your site and allowing admine also excess the site i think this what it is saying
- Thanks
{{ DiscussionBoard.errors[9792593].message }}
BenJackson 11 years ago

That is telling all bots crawling your site (including googlebot) to crawl all pages of your site. The second line is unnecessary and actually incorrect.

Here's a decent configuration for a robots.txt file on a WordPress site:

User-agent: *
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Allow: /wp-content/uploads/

You can replace your existing robots.txt (except for the sitemap!) with the above.
- Thanks
{{ DiscussionBoard.errors[9793195].message }}
SEO Power 11 years ago

Originally Posted by jsam757

what it mean can anybody plz tell...
User-agent: *
allow: /wp-admin/

Sitemap: http://www.mydomain.com/sitemap.xml

It should be disallow, not allow. If you upload it that way, Google will crawl your admin login page and the chances of your site getting hacked will increase.
- Thanks
- 1 reply
Signature
ONE HOUR INDEXING - STOP WASTING MONEY ON LINKS GOOGLE NEVER FINDS
{{ DiscussionBoard.errors[9793266].message }}
- BenJackson 11 years ago
  
  Originally Posted by SEO Power
  
  If you upload it that way, Google will crawl your admin login page and the chances of your site getting hacked will increase.
  
  No, that's simply not true.
  
  Thanks
  
  {{ DiscussionBoard.errors[9793631].message }}
yukon Banned 11 years ago

Lol, why would you even want a login page indexed?
- Thanks
{{ DiscussionBoard.errors[9793728].message }}