3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

Robots query

by danielthomas
2 replies
  • SEO
    • |
User-agent: Googlebot
Allow: / User-agent:
Googlebot-Image Allow: /
User-agent:
Googlebot-Video Allow: /
User-agent: Bingbot Allow: /
User-agent: * Disallow: www.test.in/*
User-agent: Baiduspider Disallow: /
User-agent: Yandex Disallow: /

What does this robots.txt convey? Can some one explain me? What this operator does
"/*" ?
#query #robots
  • Profile picture of the author Punit12
    Dear Daniel,

    Let me put your robots code in this way,

    Code:
    User-agent: Googlebot 
Allow: / (Let Googlebot crawl the site)
User-agent: Googlebot-Image 
Allow: / (Let Googlebot-Image crawl the site)
User-agent: Googlebot-Video 
Allow: / (Let Googlebot-Video Crawl the site)
User-agent: Bingbot 
Allow: / (Let Bingbot crawl the site)
User-agent: * 
Disallow: www.test.in/* 
User-agent: Baiduspider 
Disallow: / (Restrict Baiduspider crawling my site)
User-agent: Yandex 
Disallow: /  (Restrict Yandex crawling my site)
    What does this robots.txt convey? Can some one explain me? What this operator does
    "/*" ?

    >> According to me, Disallow: www.test.in/* is Commmand for all agent not to crawl any pages from www.test.in, but not sure if that would work.
    Signature

    Punit Kansara
    Seo Strategist
    {{ DiscussionBoard.errors[7796943].message }}
  • Profile picture of the author MonitorScout
    Code:
    User-agent: * Disallow: www.test.in/*
    The above mention string is wrong.

    This is not the way to tell search engine bots to not crawl a desired page, folder or site.

    Check the following example:
    User-agent: * Disallow: / The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.



    To allow all robots complete access
    User-agent: *

    That's all.
    Signature
    Monitor Scout - Website & Server Monitoring
    50 different checks, SNMP monitoring and much more.
    https://www.monitorscout.com
    {{ DiscussionBoard.errors[7797021].message }}

Trending Topics