Collecting ideas. What would you do with a web crawler?
Hi,
I just registered here to get some inspiration for new projects and share some of my ideas ;-)
I recently played around with web crawlers. I took an open source crawler and extended its functionality and let him crawl some websites. I crawled the about-pages of the websites and extracted the postal address (city, street + number, company name, email address, phone number, fax number) just for fun.
Both the crawler and the address parser worked well.
But what to do with the collected information?
My ideas were: Build a web directory (like yellow pages) for a special topic and let the web crawler fill this directory.
What I do NOT want: Collect data and sell it. I don't want to get involved in the spam industry
I think data mining is still promising and has high potential.
What do you think?
jim
I just registered here to get some inspiration for new projects and share some of my ideas ;-)
I recently played around with web crawlers. I took an open source crawler and extended its functionality and let him crawl some websites. I crawled the about-pages of the websites and extracted the postal address (city, street + number, company name, email address, phone number, fax number) just for fun.
Both the crawler and the address parser worked well.
But what to do with the collected information?
My ideas were: Build a web directory (like yellow pages) for a special topic and let the web crawler fill this directory.
What I do NOT want: Collect data and sell it. I don't want to get involved in the spam industry
I think data mining is still promising and has high potential.
What do you think?
jim