Go Back   WarriorForum - Internet Marketing Forums > Warrior Support Forums > Website Design
Register Blogs FAQ Social Groups CalendarHelp Desk

Reply
 
Share
LinkBack Thread Tools
Old 09-27-2009, 08:58 PM   #1
Work hard to play hard
War Room Member
 
Join Date: Aug 2007
Location: Michigan, USA.
Posts: 1,226
Thanks: 114
Thanked 232 Times in 165 Posts
Contact Info
Send a message via AIM to MarkR
Default Site crawler tool to define internal linking structure?

OK, either I'm having a mental block remembering a tool like this, or this is the first time I've ever needed this tool.

I'm inheriting support/development responsibilities for a very large, very poorly designed, 5+ year old, PHP website, with a ton of broken links (dumb move, I know!).

Is there a tool out there that crawls the site, and defines all of the internal page links? e.g. What pages link to what pages? This site is a mess and I can't see trying to manually research/define how to navigate to hundreds of pages.

Something that creates a graphical site structure would be awesome, but a text-based report that is searchable would work to.

Mark

MarkR is offline   Reply With Quote
Old 09-29-2009, 09:15 PM   #2
Work hard to play hard
War Room Member
 
Join Date: Aug 2007
Location: Michigan, USA.
Posts: 1,226
Thanks: 114
Thanked 232 Times in 165 Posts
Contact Info
Send a message via AIM to MarkR
Default Re: Site crawler tool to define internal linking structure?

So, did I stump Warriors? Is there nothing like this?

Xenu comes kind of close, but doesn't show all links off of a page.

I don't think I'm the only person that ever needed this functionality, am I?

MarkR is offline   Reply With Quote
Old 11-22-2009, 06:48 PM   #3
Active Warrior
War Room Member
 
Join Date: Sep 2009
Posts: 34
Thanks: 7
Thanked 1 Time in 1 Post
Contact Info
Default Re: Site crawler tool to define internal linking structure?

I need something like this too bro. You could try the one I'm using called OutWit Hub. It's a Firefox Plugin that crawls sites and exports to .csv. You can find it here and it's free: Harvest The Web - OutWit Technologies.

What I'm looking for is a crawler that will follow links and pull product titles. Do you know of anything like that?
davidmerrick is offline   Reply With Quote
Old 11-22-2009, 07:27 PM   #4
Sign up Bonuses
 
signupmakemoney's Avatar
 
Join Date: Nov 2009
Location: New Orleans, LA
Posts: 27
Thanks: 1
Thanked 0 Times in 0 Posts
Social Networking View Member's Myspace Profile  View Member's FaceBook Profile  View Member's Twitter Profile  View Member's YouTube Profile
Contact Info
Send a message via Yahoo to signupmakemoney
Default Re: Site crawler tool to define internal linking structure?

Google Webmaster Tools will tell you all the broken links there are in the site. You just have to verify that you own the site first.

signupmakemoney is offline   Reply With Quote
Old 11-22-2009, 07:53 PM   #5
Work hard to play hard
War Room Member
 
Join Date: Aug 2007
Location: Michigan, USA.
Posts: 1,226
Thanks: 114
Thanked 232 Times in 165 Posts
Contact Info
Send a message via AIM to MarkR
Default Re: Site crawler tool to define internal linking structure?

David,

Thanks. Outwit seems to only spider the links off the home page, one link deep, unless I'm using it wrong. I want to spider the home page, then the links the pages the home page linked to, then those page's links and so on.

signupmakemoney,

I'm not looking for just broken links (xenu does that well), I'm looking for a tool that 1) spiders the whole site, and 2) tells me what pages link to what pages, and 3) puts it in a graphical or text-based report that I can interrogate. So, when I need to find out what pages link to pagename123.html, I can find that information.

This seems like a logical request, but nothing turns up in searches.

MarkR is offline   Reply With Quote
Old 11-30-2011, 09:24 AM   #6
Warrior Member
War Room Member
 
Join Date: Dec 2009
Posts: 8
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Site crawler tool to define internal linking structure?

I need this as well...

Have you tried downloading the whole site to dreamweaver or front page and looking from these softwares?
DAN11 is offline   Reply With Quote
Old 11-30-2011, 10:50 AM   #7
... is not this chick.
 
mojojuju's Avatar
 
Join Date: Aug 2005
Location: Potemkin Village
Posts: 1,690
Thanks: 704
Thanked 739 Times in 487 Posts
Social Networking View Member's Twitter Profile 
Default Re: Site crawler tool to define internal linking structure?

Here's how you can do it with Xenu and Graphviz How to Visualize Link Flow Within a Website | WordStream
mojojuju is offline   Reply With Quote
Old 11-30-2011, 11:35 AM   #8
Work hard to play hard
War Room Member
 
Join Date: Aug 2007
Location: Michigan, USA.
Posts: 1,226
Thanks: 114
Thanked 232 Times in 165 Posts
Contact Info
Send a message via AIM to MarkR
Default Re: Site crawler tool to define internal linking structure?

Thanks for that. It seems to be what I need. I wonder how it looks and how long it takes with a site of 4k-6k pages? LOL

MarkR is offline   Reply With Quote
Old 12-01-2011, 09:37 AM   #9
Warrior Member
War Room Member
 
Join Date: Dec 2009
Posts: 8
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Site crawler tool to define internal linking structure?

Hi,
Thanks for that but I don't see where in the Xenu I can export as the requested file format?
DAN11 is offline   Reply With Quote
Reply

  WarriorForum - Internet Marketing Forums > Warrior Support Forums > Website Design

Bookmarks

Tags
crawler, define, internal, linking, site, structure, tool

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off



All times are GMT -6. The time now is 03:33 PM.