Spider Prevention Code

by 4 replies
5
HI Guys,

Can any one tell me the code you add to a web page which prevents that specific page from ever being cached and spidered by the SE's.

I believe it's a series of meta tags that have to be are added.

This is specifically to protect a product download page from every being ranked / displayed in the search results.

Thanks

--Glen
#programming #code #prevention #spider
  • <meta name="robots" content="noindex" />

    http header tag
    X-Robots-Tag: noindex

    <meta name="robots" content="noarchive" />

    <meta name="googlebot" content="nosnippet" />

    don't forget robots.txt
    • [1] reply
    • Thanks for that johnnyN,

      So do I put..

      <meta name="robots" content="noindex" />

      <meta name="robots" content="noarchive" />

      <meta name="googlebot" content="nosnippet" />

      inside the <head> </head> tags?


      I just wasn't to car on what you described

      -Gle
      • [1] reply
  • I think this might explain it better:

    The Web Robots Pages

    Essentially you need a file called robots.txt to direct spiders away from content you don't want scanned.

    Bill
Join the discussion

Next Topics on Trending Feed