8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

Working with real-time APIs

by metallic07039
7 replies
Hi,

I am building an online coupon aggregator web site. All of the content is downloaded through various APIs. Here is the challenge:

- There will be at least 20 APIs that I need to integrate.
- None of the APIs allow for caching results into a database.
- Each API query will have about 150 unique result variations (based on user's search on my site).
- Running all 20 APIs when the user searches my site will create too many usability issues.

How can I overcome this? Perhaps there is some sort of framework that allows for memory caches or something that can do the "back office" work if I do decide to develop my site to run all of these APIs for every user search?

Any other ideas?
#apis #realtime #working
  • Profile picture of the author mojojuju
    Originally Posted by metallic07039 View Post

    - None of the APIs allow for caching results into a database.
    Do the terms of service of these APIs explicitly disallow caching their results in a database?
    Signature

    :)

    {{ DiscussionBoard.errors[4472393].message }}
    • Profile picture of the author metallic07039
      Originally Posted by mojojuju View Post

      Do the terms of service of these APIs explicitly disallow caching their results in a database?
      Yes, most disallow it. Besides, it is probably good not to cache them all the time since some of the data that the APIs give is constantly changing. I am considering cache it anyway, but automatically renewing it once an hour through a Cron Job.
      {{ DiscussionBoard.errors[4472404].message }}
  • Profile picture of the author mojojuju
    That's a real bummer. I agree that caching it and refreshing it with a cron job is the best way to go. Still, it would be much better if they allowed you to do that.
    Signature

    :)

    {{ DiscussionBoard.errors[4472432].message }}
  • Profile picture of the author dudeontheweb
    I've only played with it a little bit, but you might want to look into output buffering.
    Signature

    Need a QR Code? Check out my QR Code Generator. It's FREE!

    {{ DiscussionBoard.errors[4473957].message }}
    • Profile picture of the author avatar9812
      The API does not allow persisting, but does it allow you to buffer the results in the memory?
      {{ DiscussionBoard.errors[4474082].message }}
  • Profile picture of the author Cwantwm
    You don't really have a choice here, you are going to have to cache the results in some way even if its just for an hour like you mentioned.

    My gut response is to use wordpress and store the data in either wp_options or the wp_*meta tables or custom table if they are not suitable, if you store it with a UNIX timestamp of when you retrieved it, you can then redo an API call only if the current time is more than an hour more than the stored timestamp.

    Wordpress has all the functionality you might need for what you describe.
    {{ DiscussionBoard.errors[4474086].message }}
    • Profile picture of the author unnatural
      I'd definitely cache it - even if for short periods of time it will make a huge difference.

      I actually have a similar application which uses about 15 different API's with thousands of calls an hour and this works like a charm.

      Here's an example of a cache class I use (FYI it's a flat-file cache not stored in database which I find is much faster).

      Before you make a query you'd try reading from the cache (you can set a timeout with each call). If the cache is still "alive" (not timed out) it will return the cached results, if it returns null just make the query to the API as usual and then re-write the cache afterwards.

      Note: The forum is stripping the variables for the functions - let me know if you want the class and I'll make it available to you.

      PHP Code:

      class cache {

              function       __construct() {
                  
                            define('CACHE_PATH', '../cache/');

              }

                /**
           * Reads the contents of a specific cache file
           * @param string  The filename of the cache file to open
           * @param integer  The time to live for the current cache file
           * @return Object The unserialised data stored within the cache file
           */
                function readCache (, )
          {
               =       null;
               =  .       '.cache';
              if (      file_exists(CACHE_PATH . )) {
                  if (      intval(filemtime(CACHE_PATH . )) + intval() > time()) {
                       =       fopen(CACHE_PATH . , 'r');
                      if () {
                           =       fread(, filesize(CACHE_PATH . ));
                           =       str_replace('O:16:"SimpleXMLElement"', 'O:8:"stdClass"', );
                          if (!empty() &&       unserialize() != null) {
                               =       unserialize();
                          }
                                fclose();
                      }
                  }
              }
              return ;
          }

                /**
           * Writes the value of a variable to a specified cache file with the option of serialising the data if it is an array or object
           * @param string  The filename of the cache file
           * @param string  The data to store in the cache file
           * @param boolean  Flag specifying whether the data should be serialised before being stored.
           */
                function writeCache (, ,  = true)
          {
              if (!      file_exists(CACHE_PATH . substr(, 0, strpos(, '/')))) {
                        mkdir(CACHE_PATH . substr(, 0, strpos(, '/')));
              }
               =  .       '.cache';
              if (!      file_exists(CACHE_PATH . )) {
                        touch(CACHE_PATH . );
              }
               =       fopen(CACHE_PATH . , 'w');
              if () {
                        fwrite(, ?serialize():);
                        fclose();
              }
          }

      } 
      {{ DiscussionBoard.errors[4476163].message }}

Trending Topics