Robots query

by 2 replies
3
User-agent: Googlebot
Allow: / User-agent:
Googlebot-Image Allow: /
User-agent:
Googlebot-Video Allow: /
User-agent: Bingbot Allow: /
User-agent: * Disallow: www.test.in/*
User-agent: Baiduspider Disallow: /
User-agent: Yandex Disallow: /

What does this robots.txt convey? Can some one explain me? What this operator does
"/*" ?
#search engine optimization #query #robots


  • Code:
    User-agent: Googlebot 
Allow: / 
User-agent: Googlebot-Image 
Allow: / 
User-agent: Googlebot-Video 
Allow: / 
User-agent: Bingbot 
Allow: / 
User-agent: * 
Disallow: www.test.in/* 
User-agent: Baiduspider 
Disallow: / 
User-agent: Yandex 
Disallow: /
    What does this robots.txt convey? Can some one explain me? What this operator does
    "/*" ?

    >> According to me, Disallow: www.test.in/* iwww.test.in, but not sure if that would work.
  • Code:
    User-agent: * Disallow: www.test.in/*
    The above mention string is wrong.

    This is not the way to tell search engine bots to not crawl a desired page, folder or site.

    Check the following example:
    User-agent: * Disallow: / The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.



    To allow all robots complete access
    User-agent: *

    That's all.
Join the discussion

Next Topics on Trending Feed