I Said I DON'T Want To Be Indexed, OKAY?!

[Emmanuel Betinis]

If you do NOT want Google to index a certain URL, or if Google
already indexed a certain URL of your site and you do not want
it to be listed in the search results, you simply
include the following piece of code on the page:

<meta name="robots" content="noindex,nofollow"/>



Okay, sounds simple enough...

But what if the file that is in Google's index that you
want removed just happens to be a .SWF file?!

You can't just "put" that code into a file like that.

Catch my drift?

[tfos4941]

If it has already been indexed you will have to contact Google directly. Ask them to remove the file as it breaches your copyright.

[wtatlas]

For files that have not been indexed could you not put them in a password protected directory?

[LastWarrior]

This one has always helped me get great position in the SERPs:

<meta name="ROBOTS" content="position,#1">

I told myself I wasn't going to show anyone that black hat trick, but I wanted to give something back.

LastWarrior



Crowd roars and gives thanks!

[theseoguys]

Originally Posted by LastWarrior

This one has always helped me get great position in the SERPs:

<meta name="ROBOTS" content="position,#1">

I told myself I wasn't going to show anyone that black hat trick, but I wanted to give something back.

LastWarrior

Thanks for that. It will save me a ton of time, and help my clients.

[Rich Struck]

I think you could use robots.txt to do this. Google it and see. That's what I'd try.

[tpw]

tell the robots.txt to ignore it, and on all links pointing to it, use the noindex, nofollow.

[Emmanuel Betinis]

Originally Posted by tpw

tell the robots.txt to ignore it, and on all links pointing to it, use the noindex, nofollow.

Hmmm...based on my extensive research that command can only be
used within a META tag on a webpage, but it would be great if I could
achieve the same thing (deindexing an already indexed page) by simply
controlling it from the robots.txt - but here's the HUGE problem with
doing that...even if it was possible:

Everyone else can view your robots.txt file - it's always recommended
to never have any type of private information on your robots.txt file,
so if what you want DEindexed from Google is something you don't
want anyone else to have access to, by putting it in your robots.txt
file you have good intentions, but then the directory/URL will be
able to be viewed by the public with a simple web browser by simply
seeing it listed in your robots.txt file.

Hmmm...

[BenoitT]

You can tell Google to not index your file by using the file robots.txt in your root folder. You put that inside:

PHP Code:

User-Agent: * 
Disallow: /folder/file.php
Disallow: /folder2/file2.swf 


And yes, it is safe to do it this way. Obviously if every crawlers can see it, everybody can see it. If you want something private for real, you need to put a password otherwise someone can find your page somewhere or just randomly. If you only want crawlers to not index it, it's the perfect way.

[Dennis Gaskill]

I haven't been in my Google Webmaster Tools Account for a while, but I think there's a way to request delisting in there on a per page basis. I'd look at that first.

If that's not there, perhaps the easiest thing to do is move the content you don't want Google to index and put up a different page in it's place that you do want Google to index.

[tpw]

I honestly think you can go into Google Webmaster Tools and tell them to de-index that page.

[Dennis Gaskill]

Originally Posted by tpw

I honestly think you can go into Google Webmaster Tools and tell them to de-index that page.

Now why didn't I think of that?

[tpw]

Originally Posted by Dennis Gaskill

Now why didn't I think of that?

We were posting same at same time. The time stamp is the same.

I bet if the 3yo were not wallering on me, I could have beat you... LOL

[yukon]

Forget robots.txt, try this code in your .htaccess file.

RewriteRule .*.(swf|mov|avi|flv)$ hxxp://redirect_here.com/ [R,NC]

You need to change the url (hxxp://redirect_here.com) to the location on the net that you want to send the person/bot that tries to hotlink your image.

I use this code on my sites to block google from adding certain image formats (jpg, gif) to Google Images. It works perfect for images, so it should also work for video (swf).

BTW, this thread would be better in the SEO forum.

[yukon]

Also If your host is Hostgator, go to your Cpanel -> Hotlink Protection, & use that page to block your specific file formats (swf, zip, jpg, etc...)

If you do this (Cpanel) it will update your .htaccess file on it's own, so you wouldn't need to do the manual update code in my last comment above.

[Dennis Gaskill]

Originally Posted by tpw

We were posting same at same time. The time stamp is the same.

I bet if the 3yo were not wallering on me, I could have beat you... LOL

For the first part of your comment...I know, but I couldn't help myself.

For the second part, I've got that puppy on my lap, making it hard to type. I'd call our obstacles about even.

[Emmanuel Betinis]

Originally Posted by BenoitT

You can tell Google to not index your file by using the file robots.txt in your root folder. You put that inside:

PHP Code:

User-Agent: * 
Disallow: /folder/file.php
Disallow: /folder2/file2.swf 


And yes, it is safe to do it this way. Obviously if every crawlers can see it, everybody can see it. If you want something private for real, you need to put a password otherwise someone can find your page somewhere or just randomly. If you only want crawlers to not index it, it's the perfect way.

Actually, your URL can still become indexed by using
that code! It happened to me! And if you do some
research you'll find that to be true.

Really! That command does not guarantee the link
will not be indexed! It merely states to not allow the
spider to crawl the page and gather information about
the link (like the description tag) when displaying it
in the SERPs. If other pages are linking to the URL
it can still be indexed (it happened to me so I found out
firsthand from experience on this one) & it will just
appear in the SERP as a filename only with no
description below it (like what happened to me).

If you really want to prevent a URL from being indexed,
or if you want an already-indexed URL to be removed from
the search index, you must insert the following code within
the heading of your page:

<meta name="robots" content="noindex,nofollow" />

...and then of course wait until the spider comes around to
visit the page again to see the updated META tag & remove
the page from it's index appropriately...

The only problem about my situation here is that
the file I want DEindexed is actually a .SWF file
and not an HTML file that I can add that code to.

(So I added it to the INDEX page of the directory pathway that
the SWF file is contained in to see, as a test, if Google will
DEindex only the INDEX page or the INDEX page as well as
all the associated files within that directory path - time will
tell)

[halmo]

Originally Posted by Emmanuel Betinis

If you do NOT want Google to index a certain URL, or if Google
already indexed a certain URL of your site and you do not want
it to be listed in the search results, you simply
include the following piece of code on the page:

<meta name="robots" content="noindex,nofollow"/>



Okay, sounds simple enough...

But what if the file that is in Google's index that you
want removed just happens to be a .SWF file?!

You can't just "put" that code into a file like that.

Catch my drift?

Originally Posted by Dennis Gaskill

I haven't been in my Google Webmaster Tools Account for a while, but I think there's a way to request delisting in there on a per page basis. I'd look at that first.

Yes, you can request the page removal in Google Webmaster Tools. I just did that a week ago, and Google removed the page within a few hours. Here are the instructions:

If you own the site

Verify your ownership of the site in Webmaster Tools.
On the Webmaster Tools home page, click the site you want.
On the Dashboard, click Site configuration in the left-hand navigation.
Click Crawler access, and then click Remove URL.
Click New removal request.
Type the URL of the page you want removed from search results (not the Google search results URL or cached page URL), and then click Continue. How to find the right URL. Note that the URL is case-sensitive—you will need to submit the URL using exactly the same characters and the same capitalization that the site uses.
Click Remove page from search results and cache.
Select the checkbox to confirm that you have completed the requirements listed in this article, and then click Submit Request.

Make sure you choose "Remove page and cache" in the dropdown.

Originally Posted by Emmanuel Betinis

Hmmm...based on my extensive research that command can only be
used within a META tag on a webpage, but it would be great if I could
achieve the same thing (deindexing an already indexed page) by simply
controlling it from the robots.txt - but here's the HUGE problem with
doing that...even if it was possible:

Everyone else can view your robots.txt file - it's always recommended
to never have any type of private information on your robots.txt file,
so if what you want DEindexed from Google is something you don't
want anyone else to have access to, by putting it in your robots.txt
file you have good intentions, but then the directory/URL will be
able to be viewed by the public with a simple web browser by simply
seeing it listed in your robots.txt file.

Hmmm...

Trying to de-index it with a robots.txt might be a challenge. Google might or might not do that.

Regarding the public nature of the robots.txt: You could create a folder for only those files that you don't want indexed, then disallow that folder in the robots.txt, so the file names are not shown in the robots.txt file, although, I don't know whether the filenames could be viewed by typing in the domainname.com/yourdisallowedfoldername.

You could also create a subdomain, and put all files under that you don't want indexed. Google handles subdomains separately from the main domain. You would put a robots.txt file under the subdomain that disallows indexing the whole subdomain. That way you don't have to state any folder names or file names in the robots.txt, so even though the robots.txt is public, nobody will know the folder names or file names under the subdomain.

The big thing about the noindex meta tag, the robots.txt, and restricting access or indexing through the .htaccess file is that they are advisory--not obligatory. Search engines are NOT obligated or required to obey the instructions in either of those files or the meta tag. Google respects, and obeys those instructions, and Yahoo does too. But there are other, and many smaller search engines that don't.

So, even if you have your file removed from Google search results, it doesn't mean that it's not in the search results of other search engines.

If you want a file protected from indexing, why not put it in a different folder in your cpanel that's not under public_html

[Emmanuel Betinis]

Originally Posted by tpw

I honestly think you can go into Google Webmaster Tools and tell them to de-index that page.

Right...

And it's here: https://www.google.com/webmasters/tools/removals

But the only problem with that is after putting in the URL and
clicking Continue...




...it doesn't seem like any of those options seem to fit. It seems as if
those options are all geared towards the non-website-owner (I accessed
this tools after already being logged in from my Webmaster Tools account).

[halmo]

Originally Posted by Emmanuel Betinis

Right...

And it's here: https://www.google.com/webmasters/tools/removals

But the only problem with that is after putting in the URL and
clicking Continue...




...it doesn't seem like any of those options seem to fit. It seems as if
those options are all geared towards the non-website-owner (I accessed
this tools after already being logged in from my Webmaster Tools account).

Emmanual, did you verify your website in Google Webmaster Tools? If not, Google might think that you are not the owner, and that's the reason you are getting that dropdown you posted here.

I did this a week ago, and Google removed the page within hours. I posted instructions right before your last post. See there. You should be able to get the page removed.

[Emmanuel Betinis]

Originally Posted by halmo

Regarding the public nature of the robots.txt: You could create a folder for only those files that you don't want indexed, then disallow that folder in the robots.txt, so the file names are not shown in the robots.txt file, although, I don't know whether the filenames could be viewed by typing in the domainname.com/yourdisallowedfoldername.

Exactly. And this is another thing I learned today. Those files actually
CAN be viewed IF you do not have an INDEX page for that directory
AND in your cPanel (in HostGator anyway) in the "Index Manager"
by default the setting is to display all the files! You must manually
change this for the domain and select "No Indexing" which will
THEN generate the "forbidden no access" page anytime someone
accesses the directory with no index page.

[halmo]

Originally Posted by Emmanuel Betinis

Exactly. And this is another thing I learned today. Those files actually
CAN be viewed IF you do not have an INDEX page for that directory
AND in your cPanel (in HostGator anyway) in the "Index Manager"
by default the setting is to display all the files! You must manually
change this for the domain and select "No Indexing" which will
THEN generate the "forbidden no access" page anytime someone
accesses the directory with no index page.

Didn't know you could do this. Thanks. You could also put a blank index file under the directory--I have done that.

[Emmanuel Betinis]

Originally Posted by halmo

Verify your ownership of the site in Webmaster Tools.
On the Webmaster Tools home page, click the site you want.
On the Dashboard, click Site configuration in the left-hand navigation.
Click Crawler access, and then click Remove URL.
Click New removal request.
Type the URL of the page you want removed from search results (not the Google search results URL or cached page URL), and then click Continue. How to find the right URL. Note that the URL is case-sensitive—you will need to submit the URL using exactly the same characters and the same capitalization that the site uses.
Click Remove page from search results and cache.
Select the checkbox to confirm that you have completed the requirements listed in this article, and then click Submit Request.

Make sure you choose "Remove page and cache" in the dropdown.

Thanks. I started to do that but then the checkbox below it...



...it sounds like I need to either place a block on robots.txt (which I'm
NOT going to do since I don't want it listed there) or place a NOINDEX
META tag in the file (which I mentioned earlier...this is a SWF file and I
can not place a META tag in a SWF file!)...

So since I can't do either one of the two, I can't check the box, thus
can't move forward with this type of DEindexing option.

Thankyou for the explanation though! It's very useful to know about
in the future if I ever need to DEindex an actual HTML page, but as for
a SWF page...perhaps I'll wait it out and see if the NOINDEX META tag
I placed in the INDEX page of the directory the SWF is contained in
will work. It'd be good to know that anyway.

[Emmanuel Betinis]

Originally Posted by halmo

Didn't know you could do this. Thanks. You could also put a blank index file under the directory--I have done that.

Yep exactly. Or you could be "super safe" and do both.

(blank index file AND change 'Index Manager' settings to 'No Indexing')

[halmo]

Originally Posted by Emmanuel Betinis

Thanks. I started to do that but then the checkbox below it...



...it sounds like I need to either place a block on robots.txt (which I'm
NOT going to do since I don't want it listed there) or place a NOINDEX
META tag in the file (which I mentioned earlier...this is a SWF file and I
can not place a META tag in a SWF file!)...

So since I can't do either one of the two, I can't check the box, thus
can't move forward with this type of DEindexing option.

Thankyou for the explanation though! It's very useful to know about
in the future if I ever need to DEindex an actual HTML page, but as for
a SWF page...perhaps I'll wait it out and see if the NOINDEX META tag
I placed in the INDEX page of the directory the SWF is contained in
will work. It'd be good to know that anyway.

Yes, you need to do that to check the box. Could you password protect your file, and then put in in the robots.txt to disallow it? It would be there for only a few hours until Google de-indexed the page, then you could remove the reference from the robots.txt. I would rather do that than having it indexed, but that's only my opinion. What are the chances that someone looks up your robots.txt in the next few hours (hopefully Google wouldnt take long), and what are the chances that someone sees the file if you leave it indexed (without any time limit)? I am not trying to talk you into it, but that's how I see it (even without password protecting it).

I have to say, I am not familiar with SWF files..

Didn't Google give another option to block it? I thought there were three (you mentioned two), but I might not remember right.

[halmo]

Originally Posted by Emmanuel Betinis

Yep exactly. Or you could be "super safe" and do both.

(blank index file AND change 'Index Manager' settings to 'No Indexing')

Yes, that's what I will do. Thanks.

And I added something about subdomains to my original post. Did you see that?

[Emmanuel Betinis]

Originally Posted by halmo

Didn't Google give another option to block it? I thought there were three (you mentioned two), but I might not remember right.

Yes, there is a third option (see the checkbox image above) and that
is that it returns a "404 error", which doesn't apply either since it does
NOT return a 404 error (it loads fine).

[halmo]

Originally Posted by Emmanuel Betinis

Thanks. I started to do that but then the checkbox below it...



...it sounds like I need to either place a block on robots.txt (which I'm
NOT going to do since I don't want it listed there) or place a NOINDEX
META tag in the file (which I mentioned earlier...this is a SWF file and I
can not place a META tag in a SWF file!)...

So since I can't do either one of the two, I can't check the box, thus
can't move forward with this type of DEindexing option.

Just realized, I think the third option I was thinking is the 404 error. (as Google states by the checkbox) If you remove the file from its current location, typing in its current URL will return a 404 error, won't it? That might just solve your problem. Easy.

[Emmanuel Betinis]

Originally Posted by halmo

Yes, that's what I will do. Thanks.

And I added something about subdomains to my original post. Did you see that?

Yes, I did, and thankyou for the suggestion, however, as mentioned
earlier in this thread, the DISALLOW command in robots.txt does NOT
prevent files from becoming indexed (only crawled).

I'll paste it again:

Originally Posted by Emmanuel Betinis

Actually, your URL can still become indexed by using
that code! It happened to me! And if you do some
research you'll find that to be true.

Really! That command does not guarantee the link
will not be indexed! It merely states to not allow the
spider to crawl the page and gather information about
the link (like the description tag) when displaying it
in the SERPs. If other pages are linking to the URL
it can still be indexed (it happened to me so I found out
firsthand from experience on this one) & it will just
appear in the SERP as a filename only with no
description below it (like what happened to me).

If you really want to prevent a URL from being indexed,
or if you want an already-indexed URL to be removed from
the search index, you must insert the following code within
the heading of your page:

<meta name="robots" content="noindex,nofollow" />

...and then of course wait until the spider comes around to
visit the page again to see the updated META tag & remove
the page from it's index appropriately...

The only problem about my situation here is that
the file I want DEindexed is actually a .SWF file
and not an HTML file that I can add that code to.

(So I added it to the INDEX page of the directory pathway that
the SWF file is contained in to see, as a test, if Google will
DEindex only the INDEX page or the INDEX page as well as
all the associated files within that directory path - time will
tell)

[Emmanuel Betinis]

Originally Posted by halmo

Just realized, I think the third option I was thinking is the 404 error. (as Google states by the checkbox) If you remove the file from its current location, typing in its current URL will return a 404 error, won't it? That might just solve your problem. Easy.

Yes, that will work, but what if I needed to keep the same location of the file?

I'm curious as to how else one would get a SWF URL removed from Google's index
without having to change the filename (which is why I'm going to wait this out a few
days to see if the META tag I put in the INDEX page of the directory the SWF file
is contained in works to remove the URL of the SWF - if it doesn't then I'll just
end up using the 404 option).

Thanks for the input!

[halmo]

Originally Posted by Emmanuel Betinis

Yes, I did, and thankyou for the suggestion, however, as mentioned
earlier in this thread, the DISALLOW command in robots.txt does NOT
prevent files from becoming indexed (only crawled).

I'll paste it again:

I would think that if it's not crawled, it's not indexed either (how could the bot get the content of the page if it's not crawled?), but I don't know. However, it has to crawl the site to see if there are any restrictions to be crawled. First goes to see if there is a sitemap, if not, it goes to see if there is a robots.txt, if not, it goes to see if there is a meta tag on each page.

But you are correct, none of these will CERTAINLY prevent indexing because, as I said, these instruction are not obligatory for the search engines.

[halmo]

Originally Posted by Emmanuel Betinis

Yes, that will work, but what if I needed to keep the same location of the file?

Thanks for the input!

You are welcome!

Then you put the file back after it gets deindexed. (But as long as it's under public_html, it might be indexed again even if you "restrict" it with robots.txt or any of the other methods because search engines are NOT obligated to obey it (as stated before). You might also want to check if Yahoo indexed it. I don't know how you would remove it from there.

You could put the file in a frame. Search engines can't read (or index) frames (as far as I know. I don't know the specifics, but you might want to do a search about it). Again, I don't know anything about SWF files, so I don't know if the frame would be appropriate.

Originally Posted by Emmanuel Betinis

I'm curious as to how else one would get a SWF URL removed from Google's index
without having to change the filename (which is why I'm going to wait this out a few
days to see if the META tag I put in the INDEX page of the directory the SWF file
is contained in works to remove the URL of the SWF - if it doesn't then I'll just
end up using the 404 option).

Thanks for the input!

I wouldn't think that the meta tag in the index page would do any good for another file, but I don't know. I thought the meta tags in a file refer only to that file. Why don't you want to remove the file from it's current location for a few hours? Are your customers accessing it?

[Emmanuel Betinis]

Originally Posted by halmo

I would think that if it's not crawled, it's not indexed either (how could the bot get the content of the page if it's not crawled?), but I don't know.

I thought the same thing at first...but...

...a page does not have to be crawled in order to be indexed! If Google
sees a URL being referenced by anchor text, it can still include the URL
in the index. It will not be crawled (the page will not be fetched) so it
will not have the description tag with it, and will appear as a URL only
(as it did in my case).

YouTube - Uncrawled URLs in search results

[Emmanuel Betinis]

Originally Posted by halmo

I wouldn't think that the meta tag in the index page would do any good for another file, but I don't know. I thought the meta tags in a file refer only to that file.

Right...but other than the Google URL Removal Form I'm not sure
of what other option I have for a SWF file.

Originally Posted by halmo

Why don't you want to remove the file from it's current location for a few hours? Are your customers accessing it?

Yes, customers are accessing it...but that'd be easy enough to change.
It's primarily because I'd like to figure out a way to remove a SWF URL
from Google's index without having to delete it's original location...just
to know how it's done in case it ever gets asked in the future or if I'm
ever in a similar situation in the future.

[halmo]

Originally Posted by Emmanuel Betinis

I thought the same thing at first...but...

...a page does not have to be crawled in order to be indexed! If Google
sees a URL being referenced by anchor text, it can still include the URL
in the index. It will not be crawled (the page will not be fetched) so it
will not have the description tag with it, and will appear as a URL only
(as it did in my case).

YouTube - Uncrawled URLs in search results

Good point.

[halmo]

Originally Posted by Emmanuel Betinis

Right...but other than the Google URL Removal Form I'm not sure
of what other option I have for a SWF file.




Yes, customers are accessing it...but that'd be easy enough to change.
It's primarily because I'd like to figure out a way to remove a SWF URL
from Google's index without having to delete it's original location...just
to know how it's done in case it ever gets asked in the future or if I'm
ever in a similar situation in the future.

If you have the file under a directory (not directly under the main domain), you could put this in the robots.txt to exclude everything under that directory (maybe Google would deindex it this way). Change "directory 1" to your directory name.:

User-agent: *
Disallow: /directory 1/

I can't think of any other way. Hope you will succeed.



Interestingly, Google just deindexed a page on my site that I didn't want deindexed. I have the follow tag in the file, and the robots.txt doesn't restrict it either. I would like to know why they did that.

[seoindiaforu]

Sometimes the robot disobeys the rule and as a result page gets indexed.For better results contact Google directly.

[Alan Petersen]

Originally Posted by BenoitT

You can tell Google to not index your file by using the file robots.txt in your root folder. You put that inside:

PHP Code:

User-Agent: * 
Disallow: /folder/file.php
Disallow: /folder2/file2.swf 


And yes, it is safe to do it this way. Obviously if every crawlers can see it, everybody can see it. If you want something private for real, you need to put a password otherwise someone can find your page somewhere or just randomly. If you only want crawlers to not index it, it's the perfect way.

And the big warning here is that there are shoddy characters out there who will type in YourURL.com/robots.txt to try and access that kind of stuff.

Sounds like common sense now but I hadn't really thought about it until Emmanuel let me know.

So if you have links like your download page or thank you page that you don't want crawled there are thieves who will look at your robots.txt file so be careful.

[hearme]

I have learned alot and just working on some of my website portion not to be indexed by google and other search engines.

[yukon]

Originally Posted by Alan Petersen

And the big warning here is that there are shoddy characters out there who will type in YourURL.com/robots.txt to try and access that kind of stuff.

Sounds like common sense now but I hadn't really thought about it until Emmanuel let me know.

So if you have links like your download page or thank you page that you don't want crawled there are thieves who will look at your robots.txt file so be careful.

That's why you use the .htaccess file, instead of robots.txt like I posted above.

[Emmanuel Betinis]

Originally Posted by yukon

That's why you use the .htaccess file, instead of robots.txt like I posted above.

Yukon,

thankyou for your willingness to help, and yes, I did see the
post you made a while back about using the .htaccess file,
but it seems as if that code you're suggesting prevents
Google from indexing certain files...not removes already-
indexed URLs.