Member only forum - can you get content crawled by engines but still restrict access?

Canuckystan · 05-04-2011, 11:07 AM

I'm building my membership site with DAP. Wondering if there is a way to have the content of the member only forum crawled/searched by search engines, have people click on results, but restrict access and advise surfers they have to join to see the rest of the content?

I'm sure I've seen this happen on a site but can't recall which one.

KirkMcD · 05-04-2011, 12:34 PM

First Click Free - Webmaster Tools Help

dv8domainsDotCom · 05-04-2011, 12:37 PM

Quote:

How to get Google to Access Password Protected Pages

Google Analytics, yes; the Googlebot that indexes your pages so users can find them when they search Google? No, or at least, not yet. At some point, Google might provide the functionality for you to enter login credentials that its bot will then use to crawl and index your password protected pages. But for now, there is no way to get the Googlebot (which crawls and indexes your pages according to your sitemap file), to access your password protected pages. Still, there is a way to get search engine credit for your pages while still requiring users to enter a password to access your premium content.

(this is copied/pasted from How to get Google to Access Password Protected Pages | Guru of Search (not affiliated, just giving credit) )

So, in general, no, not easy.
Feasibly, there could be ways to allow access based on referrer but this could be challenging (involve modifying .htaccess for starters...) possibly check for any plugins that might handle this for you? I wouldn't know what specifically to suggest beyond that.
Good luck Canuckystan

Canuckystan · 05-04-2011, 01:49 PM

Thanks for the replies. I'll give the old college try and see what happens.

If I have any success with this I'll post a solution.

marciayudkin · 05-05-2011, 04:11 AM

Membergate, which is a popular software package for member sites, appears to enable indexing of member content by making the article and forum post titles to be visible outside of the member section. However, someone finding the content through Google cannot read the actual content, they will get a page asking them to subscribe.

I was a member of one site where I was startled to see something I had posted in the member section come up in Google. To me, this was an unwelcome thing because I had assumed the member section was restricted only to members.

Marcia Yudkin

Sitemapper · 05-05-2011, 07:37 AM

If you want to create e.g. a HTML sitemap of password protected pages, then A1SG can probably do it (I say probably because a *few* websites cause problems with the A1SG login mechanism)

In general: You can try allow all googlebot/bing/etc. IPs access to your content combined with a FirstClick solution. I would do that if I were you... Or maybe make a custom public frontend to your private sections? (Like showing a paragraph or two?)

Canuckystan · 05-05-2011, 03:10 PM

Yup good suggestions. I'll try them all and see what works.

05-04-2011, 11:07 AM	#1
Canuckystan HyperActive Warrior Join Date: Feb 2011 Posts: 150 Thanks: 0 Thanked 26 Times in 22 Posts	Member only forum - can you get content crawled by engines but still restrict access? I'm building my membership site with DAP. Wondering if there is a way to have the content of the member only forum crawled/searched by search engines, have people click on results, but restrict access and advise surfers they have to join to see the rest of the content? I'm sure I've seen this happen on a site but can't recall which one.

05-04-2011, 12:34 PM	#2
KirkMcD Senior Warrior Member War Room Member Join Date: Apr 2006 Location: , , USA. Posts: 2,493 Thanks: 43 Thanked 168 Times in 158 Posts	Re: Member only forum - can you get content crawled by engines but still restrict access? First Click Free - Webmaster Tools Help
	Mario Badescu Skin Care Portuguese Lessons in NYC

05-04-2011, 01:49 PM	#4
Canuckystan HyperActive Warrior Join Date: Feb 2011 Posts: 150 Thanks: 0 Thanked 26 Times in 22 Posts	Re: Member only forum - can you get content crawled by engines but still restrict access? Thanks for the replies. I'll give the old college try and see what happens. If I have any success with this I'll post a solution.

05-05-2011, 04:11 AM	#5
marciayudkin Marketing Mentor War Room Member Join Date: Feb 2008 Location: Maui and Massachusetts Posts: 1,225 Thanks: 138 Thanked 643 Times in 304 Posts Social Networking	Re: Member only forum - can you get content crawled by engines but still restrict access? Membergate, which is a popular software package for member sites, appears to enable indexing of member content by making the article and forum post titles to be visible outside of the member section. However, someone finding the content through Google cannot read the actual content, they will get a page asking them to subscribe. I was a member of one site where I was startled to see something I had posted in the member section come up in Google. To me, this was an unwelcome thing because I had assumed the member section was restricted only to members. Marcia Yudkin
	Author, Meatier Marketing Copy, available in paperback, Kindle, Nook, Audible audiobook “There are few genuine thought leaders in the field of copywriting. Marcia Yudkin is one of them. The strategies she presents in Meatier Marketing Copy are all easy to understand and implement, yet profoundly insightful. If you want to write marketing copy that sizzles and sells, this book is a must-read.” - Steve Slaunwhite, Author, Start & Run a Copywriting Business, Co-Author, The Wealthy Freelancer

05-05-2011, 07:37 AM	#6
Sitemapper Programmer War Room Member Join Date: Nov 2007 Posts: 158 Thanks: 0 Thanked 5 Times in 5 Posts Social Networking	Re: Member only forum - can you get content crawled by engines but still restrict access? If you want to create e.g. a HTML sitemap of password protected pages, then A1SG can probably do it (I say probably because a few websites cause problems with the A1SG login mechanism) In general: You can try allow all googlebot/bing/etc. IPs access to your content combined with a FirstClick solution. I would do that if I were you... Or maybe make a custom public frontend to your private sections? (Like showing a paragraph or two?)
	WebHelpForums.com \| Sitemap Generator - The WSO \| Site and Link Analyzer \| SEO and PPC Tools

05-05-2011, 03:10 PM	#7
Canuckystan HyperActive Warrior Join Date: Feb 2011 Posts: 150 Thanks: 0 Thanked 26 Times in 22 Posts	Re: Member only forum - can you get content crawled by engines but still restrict access? Yup good suggestions. I'll try them all and see what works.