3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

How to extract URLs from a site (without bringing the server down!)

by blinkseo
2 replies
Hi everybody.

One of my clients is migrating to a new ecommerce platform, and we need to get a list of urls from the existing site to start mapping out the 301 redirects. Usually, I'd use a tool like Xenu or Integrity to crawl and output a list.

However, the database and server setup is so bad that it can't handle the requests from these tools and it sends the site down. This, unsurprisingly, is one of the reasons for the migration.

Does anybody know of a way to get a full list of urls without having to make a bunch of http requests which will kill the site? Any advice would be much appreciated!
#bringing #extract #server #site #urls
  • Profile picture of the author kashem
    I think any programmer can help you . He can prepare a list of urls from the database . Try to hire someone from odesk or similar site or in this forum who can do the job . It will not take too long if everything is okay . I m sure the site generates the url following a pattern . So it will not be a big problem .
    Signature
    Niche Masterly - Get Revenue Generating Keywords For Your Niche Site.
    {{ DiscussionBoard.errors[5025041].message }}
  • Profile picture of the author wasabi20
    HI there, you can do a custom "bot" who track every link with preg_match

    Thanks
    {{ DiscussionBoard.errors[5025534].message }}

Trending Topics