Dead/Expired Domain finder - Xenu alternative

3 replies
Hi,

I am planning to write a link checker to find valuable expired/dead domains by crawling the web.

Basically it works like this:
- You enter a list of seed urls to start the crawl process.
- Configure the crawler (set a proxy list, number of threads, crawl depth, etc.)
- If the crawler finds a dead link on a website it will save it to the database for later evaluation (by using moz or other APIs that check the value of a link)
- You check the evaluated list of expired domains and register the best domains to sell or use them for your own projects.

I am uncertain how I should design the crawler. I have the following ideas:
- Simple .exe that runs on your Windows PC.
- A webservice that runs on your Windows PC or a hosted VPS. You can use a web frontend with your PC/Tablet/Smartphone to command the crawler and view the results.

Please tell me if you have ideas for the crawler design or cool features.

Thanks,
jim
#alternative #dead or expired #domain #finder #xenu
  • Profile picture of the author HN
    Banned
    Originally Posted by jimjones View Post

    (by using moz or other APIs that check the value of a link)
    - You check the evaluated list of expired domains and register the best domains to sell or use them for your own projects.

    I am uncertain how I should design the crawler. I have the following ideas:
    I think this is a wrong approach. Since you intend to check the mozrank, I believe you want to find domains with high PA, DA, Mozrank.

    Why not just download the list of top 10,000 or 100,000 domain names and check whois for expiration dates. I've done just that. I 've got a database of 1.3 million domain names with moz rank info and all the details. But I haven't updated for 2 years.

    I ran it on localhost. You can simply do the html refresh as soon as it has fetched info on one domain and immediately run the script again. Or run several instances on the same page in iframes. Or open several tabs in the browser and run the crawler in each tab. Don't forget to set_time_limit() to a few seconds. I know nothing about programming desktop applications, so this is how I did it.

    P.S. Finding expired or about to expire domains isn't difficult. Will you be able to snap them before everyone else?
    {{ DiscussionBoard.errors[10283387].message }}
    • Profile picture of the author jimjones
      Being the first who registers an expired domain is the most difficult task, that's true.

      I have read in various blog comments that people are still using tools like Xenu. And there are also really pricey tools doing the job of finding expired domains by crawling the web. like this one: www*domainmining*co

      i saw the opportunity to write a better Xenu.
      {{ DiscussionBoard.errors[10283472].message }}
  • Profile picture of the author mojojuju
    [DELETED]
    {{ DiscussionBoard.errors[10283685].message }}
    • Profile picture of the author jimjones
      Originally Posted by mojojuju View Post

      I haven't use Xenu in a while. What are some of the limitations of using Xenu and taking the broken links, then feeding them to a script that checks for the availability of those domains?
      Its not possible or use a list of proxies while crawling. Especially when you use multiple threads, some websites will ban your ip fast.
      {{ DiscussionBoard.errors[10285141].message }}

Trending Topics