URL Harvesting tool

by dracoboar Posted: 14 years ago 7 replies

Hello,

I have tools that auto bookmark url's that I feed it however as my syndication network grows grabbing these url's manually is taking more time then it is worth.

Can anyone recommend a tool that will harvest urls for me. Ideally I will point it to a site and it will output a list of urls it has not seen before.

Thank you in advance.

#search engine optimization #harvesting #tool #url

janeiro82 14 years ago

I'm not quite sure what you are talkig about... maybe "scrapebox" could help?
- [1] reply
- dracoboar 14 years ago
  
  basically I am looking for a program that will crawl a website and output(like to an excel file) urls that is has not seen before.
  
  For instance a few of my articles auto syndicate to a website, i then run the url harvester and it creates an excel file with all the new urls on that website, which I can then feed to my bookmarking tool.
  
  Thanks again
kingtana1 14 years ago

Scrapebox can harvest all pages of a website. you will want to place the domain name like this into the keywords field: site:domain.com

You would be harvesting the urls from a search engine, set the max connections for that engine to 5, set time out settings to 30 seconds, tick box to use proxies, set results to 1000, click Harvest.
- [1] reply
- yukon Banned 14 years ago
  
  The downside of that (site:domain) is, If Google doesn't know about the pages neither will Scrapebox.
  
  [ 1 ] Thanks
  
  [1] reply
yukon Banned 14 years ago

I haven't tried this, might help?

Find broken links on your site with Xenu's Link Sleuth (TM)
JSProjects 14 years ago

If the site has a xml sitemap Scrapebox can grab all of the URLs from it.

Join the discussion

Next Topics on Trending Feed

9

URL Harvesting tool

dracoboar • 14 years ago • 7 replies

Hello, I have tools that auto bookmark url's that I feed it however as my syndication network grows grabbing these url's manually is taking more time then it is worth.