URL Harvesting tool

by 7 replies
9
Hello,

I have tools that auto bookmark url's that I feed it however as my syndication network grows grabbing these url's manually is taking more time then it is worth.

Can anyone recommend a tool that will harvest urls for me. Ideally I will point it to a site and it will output a list of urls it has not seen before.

Thank you in advance.
#search engine optimization #harvesting #tool #url
  • I'm not quite sure what you are talkig about... maybe "scrapebox" could help?
    • [1] reply

    • basically I am looking for a program that will crawl a website and output(like to an excel file) urls that is has not seen before.

      For instance a few of my articles auto syndicate to a website, i then run the url harvester and it creates an excel file with all the new urls on that website, which I can then feed to my bookmarking tool.


      Thanks again
  • Scrapebox can harvest all pages of a website. you will want to place the domain name like this into the keywords field: site:domain.com

    You would be harvesting the urls from a search engine, set the max connections for that engine to 5, set time out settings to 30 seconds, tick box to use proxies, set results to 1000, click Harvest.
    • [1] reply
    • Banned
      The downside of that (site:domain) is, If Google doesn't know about the pages neither will Scrapebox.
      • [ 1 ] Thanks
      • [1] reply
  • Banned
  • If the site has a xml sitemap Scrapebox can grab all of the URLs from it.

Next Topics on Trending Feed

  • 9

    Hello, I have tools that auto bookmark url's that I feed it however as my syndication network grows grabbing these url's manually is taking more time then it is worth.