Freelancer.com

Go Back   Warrior Forum - The #1 Internet Marketing Forum & Marketplace > Warrior Support Forums > Website Design
Register Blogs Social Groups CalendarHelp Desk

Reply
LinkBack Thread Tools
Unread 27th September 2009, 08:58 PM   #1
Work hard to play hard
War Room Member
 
Join Date: 2007
Location: Michigan, USA.
Posts: 1,240
Thanks: 113
Thanked 233 Times in 166 Posts
Default Site crawler tool to define internal linking structure?

OK, either I'm having a mental block remembering a tool like this, or this is the first time I've ever needed this tool.

I'm inheriting support/development responsibilities for a very large, very poorly designed, 5+ year old, PHP website, with a ton of broken links (dumb move, I know!).

Is there a tool out there that crawls the site, and defines all of the internal page links? e.g. What pages link to what pages? This site is a mess and I can't see trying to manually research/define how to navigate to hundreds of pages.

Something that creates a graphical site structure would be awesome, but a text-based report that is searchable would work to.

Mark

MarkR is offline   Reply With Quote
Unread 29th September 2009, 09:15 PM   #2
Work hard to play hard
War Room Member
 
Join Date: 2007
Location: Michigan, USA.
Posts: 1,240
Thanks: 113
Thanked 233 Times in 166 Posts
Default Re: Site crawler tool to define internal linking structure?

So, did I stump Warriors? Is there nothing like this?

Xenu comes kind of close, but doesn't show all links off of a page.

I don't think I'm the only person that ever needed this functionality, am I?

MarkR is offline   Reply With Quote
Unread 22nd November 2009, 07:48 PM   #3
Active Warrior
War Room Member
 
Join Date: 2009
Posts: 32
Thanks: 6
Thanked 1 Time in 1 Post
Default Re: Site crawler tool to define internal linking structure?

I need something like this too bro. You could try the one I'm using called OutWit Hub. It's a Firefox Plugin that crawls sites and exports to .csv. You can find it here and it's free: Harvest The Web - OutWit Technologies.

What I'm looking for is a crawler that will follow links and pull product titles. Do you know of anything like that?
davidmerrick is offline   Reply With Quote
Unread 22nd November 2009, 08:27 PM   #4
Sign up Bonuses
 
signupmakemoney's Avatar
 
Join Date: 2009
Location: New Orleans, LA
Posts: 27
Thanks: 1
Thanked 0 Times in 0 Posts
Default Re: Site crawler tool to define internal linking structure?

Google Webmaster Tools will tell you all the broken links there are in the site. You just have to verify that you own the site first.

signupmakemoney is offline   Reply With Quote
Unread 22nd November 2009, 08:53 PM   #5
Work hard to play hard
War Room Member
 
Join Date: 2007
Location: Michigan, USA.
Posts: 1,240
Thanks: 113
Thanked 233 Times in 166 Posts
Default Re: Site crawler tool to define internal linking structure?

David,

Thanks. Outwit seems to only spider the links off the home page, one link deep, unless I'm using it wrong. I want to spider the home page, then the links the pages the home page linked to, then those page's links and so on.

signupmakemoney,

I'm not looking for just broken links (xenu does that well), I'm looking for a tool that 1) spiders the whole site, and 2) tells me what pages link to what pages, and 3) puts it in a graphical or text-based report that I can interrogate. So, when I need to find out what pages link to pagename123.html, I can find that information.

This seems like a logical request, but nothing turns up in searches.

MarkR is offline   Reply With Quote
Unread 30th November 2011, 10:24 AM   #6
Warrior Member
War Room Member
 
Join Date: 2009
Posts: 8
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Site crawler tool to define internal linking structure?

I need this as well...

Have you tried downloading the whole site to dreamweaver or front page and looking from these softwares?
DAN11 is offline   Reply With Quote
Unread 30th November 2011, 11:50 AM   #7
... is not this chick.
 
mojojuju's Avatar
 
Join Date: 2005
Location: Potemkin Village
Posts: 1,741
Thanks: 743
Thanked 768 Times in 506 Posts
Default Re: Site crawler tool to define internal linking structure?

Here's how you can do it with Xenu and Graphviz How to Visualize Link Flow Within a Website | WordStream
mojojuju is online now   Reply With Quote
Unread 30th November 2011, 12:35 PM   #8
Work hard to play hard
War Room Member
 
Join Date: 2007
Location: Michigan, USA.
Posts: 1,240
Thanks: 113
Thanked 233 Times in 166 Posts
Default Re: Site crawler tool to define internal linking structure?

Thanks for that. It seems to be what I need. I wonder how it looks and how long it takes with a site of 4k-6k pages? LOL

MarkR is offline   Reply With Quote
Unread 1st December 2011, 10:37 AM   #9
Warrior Member
War Room Member
 
Join Date: 2009
Posts: 8
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Site crawler tool to define internal linking structure?

Hi,
Thanks for that but I don't see where in the Xenu I can export as the requested file format?
DAN11 is offline   Reply With Quote
Reply

  Warrior Forum - The #1 Internet Marketing Forum & Marketplace > Warrior Support Forums > Website Design

Bookmarks

Tags
crawler, define, internal, linking, site, structure, tool

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




All times are GMT -6. The time now is 01:29 AM.