How to keep pages from being indexed

by gjd6599

Posted: 15 years ago 6 replies

PROGRAMMING

Hi, I have some pages being indexed that I don't want showing on the internet. These are product download sites, that customers should only get to after paying for the product. A warrior gave me a code today to use. I want to know if I should take the other items out also. The top code is what I was given. The bottom code is from my site. Should I also remove stuff like front page editor, and content type, and content language, or will that cause problems. Thanks Glenn

<html>
<head>
<meta name="robots" content="noindex" />
<title>Don't index this page</title>
</head>

<meta name="robots" content="noindex"/>
<meta name="ProgId" content="FrontPage.Editor.Document">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<meta http-equiv="Content-Language" content="en-us">
<title>Don't index this page</title>

#indexed #pages

wayfarer 15 years ago

The noindex meta tag is correct. I would definitely remove the frontpage tag, because it alerts hackers to potential vulnerabilities on your site (in certain cases, Frontpage sites are hackable, and in a really easy and evil way, so there is no reason to alert people that you are using it).

Which brings me to the point: why in the world are you using Frontpage? Did you know that Microsoft discontinued this product, because of how bad it is? It tends to produce garbage code, with lots of proprietary Microsoft stuff, that only works in IE.
- Thanks
Signature
I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
{{ DiscussionBoard.errors[223983].message }}
nrocy 15 years ago
You only need the meta tag with the name "robots":
Code:

<meta name="robots" content="noindex" />
I'd probably also add "nofollow" (which tells the spider to ignore links on the page too):
Code:

<meta name="robots" content="noindex, nofollow" />
The other meta tags that you have will not stop the robots one from working in any way.

Worth noting that not all robots will comply, so you'd be better off securing the pages behind a login if possible.

- Paul
- Thanks
- 1 reply
{{ DiscussionBoard.errors[224010].message }}
- gjd6599 15 years ago
  
  Thanks for the help. In answer to wayfarer's question, I bought front page 2002 when it first came out, and got pretty good at it. It was easy for me to learn because it is similar to microsoft word which I also use a lot.
  
  I have been too lazy to learn another program like dreamweaver which I actually own. I also have several sites, and it would be a pain to switch over. My sites are simple and basic. Am I hurting myself in the search engines with frontpage?
  
  Thanks Glenn
  
  Thanks
  
  {{ DiscussionBoard.errors[224177].message }}
awesometbn 15 years ago

Doesn't FrontPage live on today by another name, like ExpressionWeb?

In addition to the inline exclusion you've done with meta tags you can also create a text file and upload it to the root directory on your web server, called robots.txt. For an excellent and detailed explanation of what you can put in this file, see the following,

SEO with Robots.txt
- Thanks
- 1 reply
{{ DiscussionBoard.errors[224394].message }}
- gjd6599 15 years ago
  
  That's a pretty cool site awesometbn. I see a few codes there I could use. Thanks for the link. Glenn
  
  Thanks
  
  1 reply
  
  {{ DiscussionBoard.errors[224987].message }}
  
  Techie Turtle 15 years ago
  
  Don't forget that anyone can go to http://YourDomain.com/robots.txt and see the list of sub-directories you're "hiding." You should take one additional step and place a blank html file in those directories and call it index.html.
  
  Just 1 more step to keep everything hidden!
  
  Thanks
  
  {{ DiscussionBoard.errors[228881].message }}