3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

Robots.txt

by ghazia
2 replies
  • SEO
    • |
User-agent: Googlebot
Disallow: Disallow: /*?
Disallow: /content/

User-agent: Bingbot Disallow:
Disallow: /*?
Disallow: /content/

User-agent: Slurp
Disallow: /
Disallow: /*?
Disallow: /content/

User-agent: *
Disallow: / Disallow: /*?
Disallow: /content/

This is my robots.txt for my site. I have set this for Google, Yahoo and bing to crawl. But my site gets listed in Lycos, duckduckgo and many other search engines.
#robotstxt
  • Profile picture of the author mastertushar
    I am using wordpress site and using this robots.txt


    User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /go/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: *?wptheme Disallow: ?comments=* Disallow: /search? Disallow: /?p=* Disallow: /ko/ Disallow: /bg/
    User-agent: Mediapartners-Google* Allow: /
    User-agent: Googlebot-Image Allow: /wp-content/uploads/
    User-agent: Adsbot-Google Allow: /
    User-agent: Googlebot-Mobile Allow: /

    But still my website is visible in yahoo
    ..

    I think it will take time for it .
    {{ DiscussionBoard.errors[5829918].message }}
  • Profile picture of the author shaadishoppe
    1. You have not mentioned when you have uploaded robots.txt file on the server.

    2. You should write each text on new lines.

    Disallow: Disallow: /*? is wrong way.

    Disallow:
    Disallow: /*? is right way.

    That might be the reason, why Search Engines can't block your specified contents. Update your robots.txt file as soon as possible.
    {{ DiscussionBoard.errors[5830010].message }}

Trending Topics