Grabbing DataFeed Prices With PHP

by 7 replies
9
I really want a script which can take prices and stock levels from multiple merchant data feeds and display this in a table.

I am reluctant to hire a programmer to do this, I want to try myself.

Can any one tell me the process I would need to go through and the type of php functions i would need to look out. If you can give me a basic outline of the procedure then I think I can get the rest.

thanks
#programming #datafeed #grabbing #php #prices
  • This should be pretty easy.

    Step 1: Write a backend script to scrape the URLs (or, if possible, RSS feeds) of the merchants' sites that have the info you want. Since you're using PHP, you can use curl for this (you'll need the curl_init, curl_setopt, and curl_exec functions). Depending on whether you're using URLs or RSS feeds, you'll probably need to use preg_match or PHP's XML parsing functions to pull out the data you want.

    Step 2: Once you've parsed out the prices and stock levels, store them in a local MySQL database (you'll need the mysql_connect, mysql_select_db, and mysql_query functions).

    Step 3: Set your backend script to run every X minutes through cron.

    Step 4: Build a web page in PHP which connects to your MySQL database, and displays an HTML table containing the data you collected (you'll need the mysql_query and mysql_fetch_assoc functions).
  • yup.. the ol' screen scrape usually works.
  • If you mean rss feeds then it's fairly easy, as described, I can add to that if I get a chance later - a library like SimplePie makes it easy. If you mean combining multiple downloadable 'feeds' (actually csv files) from different merchants then it's a pig. There are product feed aggregation services out there, I recommend using one, doing it yourself, well, that way lies madness, I can explain that in more detail at some point when I'm not on my mobile (yes, I even follow WF on my mobile, tragic!)
    • [1] reply
    • Jeez.....after a few late nights I finally completed it!

      Having no knowledge of mysql and php, I feel really pleased with myself. I paid for online tuturials in the essentials which cost me $25, but it was worth it because I have learned a lot.

      Basically I can not take a product datafeed from a network, upload it, use a set of drop down boxes to assign the CSV columns to the field in the table and then import.

      I did this by building a function to make the drop down boxes in html form, which displayed the column header in the csv file, but when its posted to the processing script it sends the position key in the array. I then use the $_post array to assign each value to a new variable which is matched to the table field.

      its getting confusing now so i will stop...but hey i did it! I am actually thinking of taking it further and building a industrial proof price comparison script!


      thanks for your earlier comments they were useful to me during my research.
  • Hey, well done for managing this. I am not sure I exactly understood your implementation, but make sure you try to break it thoroughly before slapping a site up there! I did a lot of this kind of stuff a while back, when I used to spend more time programming. I will tell you what problems I had:

    - you want to automate the whole process - the datafeeds get updated daily or weekly, right?
    - you need to be careful you are not maxing out your CPU time - importing 50,000 records or whatever, just like that, can take a looong while, especially if you are doing a lot of processing too. Look into LOAD DATA INFILE - very fast though not very flexible.
    - aggreggating datafeeds from more than one source is a nightmare. Even the datafeeds from one network, like CJ, are all COMPLETELY different, it's chaos. So you have to harmonise the data somehow. Again, are you going to automate this in some way?
    - are you indexing the data in some way? Say if each product has, I dunno, a colour specified. It would be much more efficient to create a new db table just with colours in, and a foreign key in the main dbase. That can also be tricky to automate.

    Not to discourage you, just to give you things to think about. It's ideal if you can automate and scale this process. You don't want to be doing manual data-shuffling every day, right!

    Think cron job, that fetches the file remotely, unpacks it, imports it, does all the other stuff, quite a bit of work, but worth it in the end.
    • [1] reply
    • Also difficult to implement on shared hosting because you never get FILE permission.
      • [1] reply
Join the discussion

Next Topics on Trending Feed