Scraping/crawling data

by 15 replies
17
Not sure if this is the correct board to post this on, but I would like to discuss algorithmic data gathering here.

Specifically, I have been more and more involved in scraping data for various clients and I've gotten quite good at it.

Recently I acquired about 1.5GB of UK property sales data for a client, and it got me thinking:

- is there an organized market where one could sell such data?
- what is the legal status of scraping? some resources seem to say it's legal because you're just gathering information that's already publicly posted, some say it's in some sort of grey area or dependent on the specific website's terms of service
- has anyone made good money through scraping and would like to share their experience?
#programming #data #scraping or crawling
  • What did you use to scrap data?
    • [3] replies
    • Python's "scrapy" library.
      • [1] reply

    • Usually it's PHP. There is nothing difficult in content scrapping. Even for people with a basic knowledge base in regular expressions.

    • You can use a Python script to scrape the web. Al Sweigart has a chapter about it in his book, Automate the Boring Stuff with Python. He also has a Youtube channel.

      You can read that chapter online for free on his website. Go to:

      https://automatetheboringstuff.com

      He has a ton of free Python training.

      John
  • What you're talking about would be lead generation. The value of the data would be in if you could analyze it to be a set of data that could convert to leads for a business that could use the data. If you can do that, you can make money.
    • [ 1 ] Thanks
    • [1] reply
    • Thanks. But how would I go about selling the leads? Is there an organized marketplace for this sort of thing, or do I just find and contact businesses directly?
      • [1] reply
  • The only glaring issue with this approach; it lacks a warm, personal, genuine, human touch. Stick with building bonds to build a thriving business, the right way. I Junked-Trashed a few more emails today from bloggers scraping my data and email, sending me unsolicited emails, Silly approach because you chase out of fear, while I attract out of love. Peep my Seen At Page to see how it has worked for me hehehehe
    • [ 1 ] Thanks
    • [1] reply
    • Dude, I'm a programmer, not a blogger or a marketer. I am looking to market my skills, not change professions.
  • There definitely is a market for scraped data and scarping services. Is it legal or illegal? Like you said, it really depends. It's widely thought of as "gray hat" or black hat by those that know nothing about it and those that profit from it don't even worry about it much lol..

    There is a lot of uses for scarped data. I suggest you search google and visit blackhat forums (no not all information on blackhat forums is shady). The more research you do on scrapping, the more you understand how to use it for profit whether using the scraped data yourself or by selling it or by performing scraping services for other businesses. We live in the information age and data is valuable.

    Businesses scrape their competitors' websites for data they can use to get a better picture of their competitor and to compete better against them. Companies with old, large websites even scrape their own websites to find specific data they need to change, remove or analyze.

    Say what you will about scarping contact info for selling to them, but it works. If it didn't no one would scrape contact info. look at it this way, why does telemarketing and spam still exist? Because a good percentage of people will buy from people selling via telemarketing and spam. People are suckers for a good "deal", then they cry when they get scammed. If people buy into scams from spammers, they will also buy legit products if "pitched" properly. But scraped contact info can be used in many ways, not just for sending emails. Salespeople can used scarped contact info when they are looking for more leads to target. Nothing wrong with it. And it beats searching for it manually. Scraping is more efficient if done properly so it saves them time, even if it costs them money.

    Lots of uses for scraped data.
    • [ 2 ] Thanks
  • In sort, it is not legal, if you sell and also depend upon To whom you are selling.
    • [1] reply
    • pectel is it legal to scrap website and use data online ?
  • You can sell it on black where opposite company do not care - from where these data is coming
    But You would get very less amount.

    You can not sell users pics, messages, emails messages and other sensitive information, even you can not sell user log.
    You can not sell browser cookies and cache files.
  • Programming is very Logical That's why it is Very DIficult
    • [ 1 ] Thanks
  • [DELETED]
  • [DELETED]

Next Topics on Trending Feed

  • 17

    Not sure if this is the correct board to post this on, but I would like to discuss algorithmic data gathering here. Specifically, I have been more and more involved in scraping data for various clients and I've gotten quite good at it.