5 {{ upvoteCount | shortNum }}
5 {{ upvoteCount | shortNum }}

Forum Scrapping - Need Advice?

by GoGetta
4 replies
Hey Warriors,

Ok, I am just looking for some advice and insight from programmers of the WF!

I have an idea for a web app that I want to develop. However, it involves scrapping forums regularly.

Now, I know forums can block apps that do this and was wondering what your thoughts are on this?? Is it worth investing and developing this?? Are there ways around this?? And is it going to cause headaches further down the line??

As far as which forums I want to scrape, let's just say for this thread, all forums! I am new to programming and web app development and need advice from others of whether this is worth investing in.

I have someone ready to work with on this that can do what I want, but I have heard a few statements from others regarding issues with the app being blocked out by forum owners quickly and easily, so need some advice??

If that makes sense! lol

GoGetta
#advice #forum #scrapping
  • Profile picture of the author Manfred Ekblad
    If you do it the right way your app wont be blocked.

    How are you going to use the scraped data?
    {{ DiscussionBoard.errors[2071619].message }}
  • Profile picture of the author GoGetta
    Hey,

    What do you mean the right way?

    The intention is to use the data to monitor for new threads!

    GoGetta
    {{ DiscussionBoard.errors[2071670].message }}
  • Profile picture of the author Manfred Ekblad
    Do you really need to scrape the forum for that?

    Most forums have functionality such as getting all the new posts http://www.warriorforum.com/search.php?do=getnew it only takes a few GET-requests every now and then to monitor for new threads then.

    The reason most scrapers gets blocked are because they cause problems for the forum owner (adding load to the server, copying all the data, spamming...). So, design your app to cause as little problems as possible.
    {{ DiscussionBoard.errors[2071695].message }}
  • Profile picture of the author senderbot
    You can get Google to do it for you too.

    Set up a Google alert and use the following format in the alert box. In this example I want to search this forum for the keyword "Wordpress" so this would be my alert:

    wordpress url:www.warriorforum.com

    I get daily emails based on the keywords of my choice. Although I would only get ones that happened to make their way onto Google.

    Cheers

    Max
    Signature
    PornStarStamina is for sale! - Buy the book rights and website! or Just Download the Book For FREE! - Check it out!
    {{ DiscussionBoard.errors[2073137].message }}

Trending Topics