HTML site advice please

15 years ago

The point is that robots will only find pages that are linked one to another, directly or indirectly.

If a page is an "island" and has no other page linking to it, it will simply never be found.

Why not put a link on every page to a site map and the site map has links to every other page on your site.

Cheers,

Neil

Thanks
2 replies

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[738784].message }}

TheRichJerksNet

15 years ago

Originally Posted by Neil Morgan

The point is that robots will only find pages that are linked one to another, directly or indirectly.

If a page is an "island" and has no other page linking to it, it will simply never be found.

Why not put a link on every page to a site map and the site map has links to every other page on your site.

Cheers,

Neil

Hi Neil,
Sorry but this is the first time I will have to disagree with you ... Google bot scans your web server and it does find pages that are not linked to anything, this is why you must use .htaccess or robots.txt file in non-password protected folders if you do not want pages indexed.

To answer the OP....

You can have a catch all page if you choose, sort of like a sitemap.. Which you should have a sitemap anyways.

James

Thanks

{{ DiscussionBoard.errors[738885].message }}

nvs74191

15 years ago

Have to disagree on this. I set up a wp site using a template that makes each page an island, and each and every page has got indexed and ranked quite quickly.

Originally Posted by Neil Morgan

The point is that robots will only find pages that are linked one to another, directly or indirectly.

If a page is an "island" and has no other page linking to it, it will simply never be found.

Why not put a link on every page to a site map and the site map has links to every other page on your site.

Cheers,

Neil

Thanks

Signature

Swaminathan

I am a Jack Duncan Fan!

{{ DiscussionBoard.errors[739422].message }}

blueclcl

15 years ago

Thanks neil, so at the bottom of every page i would have all reviews which would then link to my site map.

Thanks

Signature

Free cheat sheet for getting your first paying writing client on Upwork within 7 days - Learn More [/B]

{{ DiscussionBoard.errors[738823].message }}

Neil Morgan

15 years ago

That sounds cool. As long as there is a path to all pages from all other pages and there are no cut-off "islands".

Cheers,

Neil

Thanks

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[738844].message }}

TheNightOwl

15 years ago

Try digging up some of Michael Campbell's early stuff -- he gives it away for free as a sign-up incentive. There's some good stuff in one of his free ebooks from Leslie Rhodes at Planet Ocean about routing the link-juice.

Thanks

Signature

Who is Gatchaman?

{{ DiscussionBoard.errors[738871].message }}

Neil Morgan

15 years ago

Google bot scans your web server and it does find pages that are not linked to anything

Hi James

Cool. OK, geek head on...

...from a technical point of view, I'd love to know how it does that because there is no "master list" of pages and no way for the directory to be scanned.

Cheers,

Neil

Thanks
2 replies

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[738891].message }}

Neil Morgan 15 years ago

Actually, it makes me want to uninstall the Google toolbar.

It MUST be getting the URL from somewhere.

Cheers,

Neil
- Thanks
Signature

Easy email marketing automation without moving your lists.
{{ DiscussionBoard.errors[738907].message }}

TheRichJerksNet

15 years ago

Originally Posted by Neil Morgan

Hi James

Cool. OK, geek head on...

...from a technical point of view, I'd love to know how it does that because there is no "master list" of pages and no way for the directory to be scanned.

Cheers,

Neil

Its a bot, it scans the entire webserver ...lol I was actually surprised myself when I had pages indexed in google that I never linked to or though google would ever see.. Ever since then I have been using .htaccess or robots.txt to block access to pages that I do not want indexed.

James

Thanks

{{ DiscussionBoard.errors[738923].message }}

Tyrus Antas

15 years ago

Google can't "scan" your entire webserver. If you have webpages not linked from anywhere, no way in hell Google can find them. If they found something, It's because there is a link somewhere or they're using user data to get there: Google webmasters central? Google Toolbar? Chrome?

Tyrus

Thanks

{{ DiscussionBoard.errors[738983].message }}

Neil Morgan

15 years ago

Hi James

Its a bot, it scans the entire webserver

...seemingly using a lot of weird and wonderful ways to find out what's there. Do a Google search on "google unliked pages" or some such. Scary stuff.

The bottom line is, if you don't want it to be found, don't put it on your web server.

We live and learn

Cheers,

Neil

Thanks

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[738994].message }}

TheRichJerksNet

15 years ago

Tyrus - I am sorry but you are wrong ... I can put up pages right now not linking to anyplace and trust me google will pick them up in 48 hours. I should know I have been doing website development for over 15 years and I have well over 500 websites on one server of mine alone...

I have trackd google's bot inside folders that are private, not password protected but private... So when they tick me off I just add a .htaccess code - Redirect 301 / Google - Which kicks google's bot back to google..

Neil - You are correct, if you do not want it to be found, then keep it offline ...

James

Thanks
1 reply

{{ DiscussionBoard.errors[739282].message }}

AllThingsArePossible

15 years ago

Originally Posted by TheRichJerksNet

Tyrus - I am sorry but you are wrong ... I can put up pages right now not linking to anyplace and trust me google will pick them up in 48 hours. I should know I have been doing website development for over 15 years and I have well over 500 websites on one server of mine alone...

I have trackd google's bot inside folders that are private, not password protected but private... So when they tick me off I just add a .htaccess code - Redirect 301 / Google - Which kicks google's bot back to google..

Neil - You are correct, if you do not want it to be found, then keep it offline ...

James

I believe your right...sometimes I find a "Googled file "in my webserver for some strange reason out of no where.

Do you ad the .htaccess in the index file or a for all pages for that particular folder?

Thanks
1 reply

Signature

peak short video - Im ready...are you?

{{ DiscussionBoard.errors[739342].message }}

TheRichJerksNet

15 years ago

Originally Posted by AllThingsArePossible

I believe your right...sometimes I find a "Googled file "in my webserver for some strange reason out of no where.

Do you ad the .htaccess in the index file or a for all pages for that particular folder?

I normally add it in the folder I am working in .. I do many websites and many times I create the sites online so clients can see the development of the sites. So I normally build the sites inside another folder other than the root and I will add the robots.txt file and or .htaccess to that folder.

Obviously I do not wwant to block google from main pages so this is why I do it on a per folder basis..

@ Neil - Hey, you offering to be my lawyer ...lol

James

Thanks

{{ DiscussionBoard.errors[739386].message }}

Neil Morgan

15 years ago

In defence of Tyrus, he's thinking along the same lines that I was and is right.

The bots do need to find a link somwhere - it's just that they find it in places where the sun don't shine.

Cheers,

Neil

Thanks
1 reply

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[739289].message }}

TheRichJerksNet

15 years ago

Originally Posted by Neil Morgan

In defence of Tyrus, he's thinking along the same lines that I was and is right.

The bots do need to find a link somwhere - it's just that they find it in places where the sun don't shine.

Cheers,

Neil

LOL ... Yeah it can be a pain sometimes... I had google hitting one of my websites 1,200 times a day for 2 weeks straight.

Thanks

{{ DiscussionBoard.errors[739299].message }}

Neil Morgan

15 years ago

Sue them for DOS

Thanks

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[739306].message }}

Neil Morgan

15 years ago

@ Neil - Hey, you offering to be my lawyer ...lol

Hang on, I'll put up a WSO

Thanks

Signature

Easy email marketing automation without moving your lists.

{{ DiscussionBoard.errors[739389].message }}

TheRichJerksNet

15 years ago

Neil,
Lawyers around here get about $375 an hour ... So WSO price would be how much ...lol Oh and do you offer a iron clad refund policy ...lol

James

Thanks

{{ DiscussionBoard.errors[739403].message }}

HTML site advice please

Trending Topics

How did you turn it all around?

Any thoughts on golf's latest phenom...Scottie Scheffler??

Infographic: How B2B marketers on LinkedIn can maximise their efforts for better audience reach

What's the Best IM-Related Skill to Learn Today That Can Help in the Future?

Some interesting stats about brand comms