5 {{ upvoteCount | shortNum }}
5 {{ upvoteCount | shortNum }}

Spider Prevention Code

by GlenH
4 replies
HI Guys,

Can any one tell me the code you add to a web page which prevents that specific page from ever being cached and spidered by the SE's.

I believe it's a series of meta tags that have to be are added.

This is specifically to protect a product download page from every being ranked / displayed in the search results.

Thanks

--Glen
#code #prevention #spider
  • Profile picture of the author johnnyN
    <meta name="robots" content="noindex" />

    http header tag
    X-Robots-Tag: noindex

    <meta name="robots" content="noarchive" />

    <meta name="googlebot" content="nosnippet" />

    don't forget robots.txt
    {{ DiscussionBoard.errors[4573835].message }}
    • Profile picture of the author GlenH
      Thanks for that johnnyN,

      So do I put..

      <meta name="robots" content="noindex" />

      <meta name="robots" content="noarchive" />

      <meta name="googlebot" content="nosnippet" />

      inside the <head> </head> tags?


      I just wasn't to car on what you described

      -Gle
      {{ DiscussionBoard.errors[4573906].message }}
      • Profile picture of the author johnnyN
        Originally Posted by GlenH View Post

        Thanks for that johnnyN,

        So do I put..

        <meta name="robots" content="noindex" />

        <meta name="robots" content="noarchive" />

        <meta name="googlebot" content="nosnippet" />

        inside the <head> </head> tags?


        I just wasn't to car on what you described

        -Gle
        yes these go in the <head></head> section
        {{ DiscussionBoard.errors[4574073].message }}
  • Profile picture of the author mywebwork
    I think this might explain it better:

    The Web Robots Pages

    Essentially you need a file called robots.txt to direct spiders away from content you don't want scanned.

    Bill
    {{ DiscussionBoard.errors[4573946].message }}

Trending Topics