Remote form submission via PHP Curl

by khtm
6 replies
Hey nerd-buddies!

I have a problem -

I need to write a bunch of interfaces to social networking websites that don't have APIs available. I've been using PHP Curl to connect and in some cases have been successful at logging in but then can't seem to submit a form on another page.

The code needs to be ran automatically via a cron job (so...behind the scenes).

Does anyone have some mad skills at simulating remote form submissions via PHP? Or should I be using javascript or another language?

Thanks!
#cron #curl #form #form submission #logging #php #remote #simulating #submitting #website
  • Profile picture of the author maxleadford
    Kane,

    It's going to depend on the site you're using. Something like Facebook runs on Javascript 99% of the time, so straight form submissions aren't going to work for you there.

    If the site is 100% non-RPC based, then you might just need to capture the login cookie(s) that gets set and make sure you reply with that header when posting new form submissions.

    Code:
    $user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0';
    $headers = array (
        'HTTP_ACCEPT' => 'application/x-shockwave-flash,text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5', 
        'HTTP_ACCEPT_LANGUAGE' => 'en-us,en;q=0.5',
        'HTTP_ACCEPT_CHARSET' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
        'HTTP_KEEP_ALIVE' => '300',
        'HTTP_CONNECTION' => 'keep-alive'
        );
        
        
    
    function read_header($ch, $string)
    {
       global $location; #keep track of location/redirects
       global $cookiearr; #store cookies here
       global $ch;
           # ^overrides the function param $ch  this is okay because 
           # we need to update the global $ch with new cookies
      
       $length = strlen($string);
       if(!strncmp($string, "Location:", 9)) #keep track of last redirect
       { $location = trim(substr($string, 9, -1)); }
       
       if(!strncmp($string, "Set-Cookie:", 11)) #get the cookie
       { 
         $cookiestr = trim(substr($string, 11, -1));
         $cookie = explode(';', $cookiestr);
         $cookie = explode('=', $cookie[0]);
         $cookiename = trim(array_shift($cookie));
         $cookiearr[$cookiename] = trim(implode('=', $cookie));
       }
       $cookie = "";
       if(trim($string) == "") #execute only at end of header
       {  
         foreach ($cookiearr as $key=>$value) { $cookie .= "$key=$value; "; }
         curl_setopt($ch, CURLOPT_COOKIE, $cookie);
       }
    
       return $length;
    }
    
    
    
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'read_header');  # take their cookie and love it!
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
    curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
    curl_setopt($ch, CURLOPT_REFERER, $url_base);
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($ch, CURLOPT_URL, $url_login);
    
    $string = curl_exec($ch);
    Set your unset variables accordingly and give it a run.

    Hope this helps!
    {{ DiscussionBoard.errors[993895].message }}
  • Profile picture of the author khtm
    Hey Max, thanks for the help!

    I think I have a lot to learn about cookies and HTTP headers...this stuff is still confusing for me.

    The first site I'm trying this on is :: BuddyMarks ::

    I wanted to first initiate a login and then insert a new bookmark. I simply viewed the form details and tried to use Curl POSTFIELDS to copy them all.

    Here's my original code:

    PHP Code:
     'myusername';
     = 
    'mypassword';

    // Post variables
     
    'form_username='.urlencode().'&form_password='.urlencode();
     .= 
    '&action=login';

    // Curl setup
     
    curl_init();
    curl_setopt(, CURLOPT_URL'http://www.buddymarks.com/login.php');
    curl_setopt(, CURLOPT_POST1);
    curl_setopt(, CURLOPT_POSTFIELDS, );
    curl_setopt(, CURLOPT_HEADER0);
    curl_setopt(, CURLOPT_FOLLOWLOCATION1);
    curl_setopt(, CURLOPT_SSL_VERIFYPEERfalse);
    curl_setopt(, CURLOPT_COOKIEJAR"my_cookies.txt");
    curl_setopt(, CURLOPT_COOKIEFILE"my_cookies.txt");
    curl_setopt(, CURLOPT_RETURNTRANSFER1);
    curl_setopt(, CURLOPT_USERAGENT"Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (.NET CLR 3.5.30729) Creative ZENcast v2.00.14");

    // Login
     
    curl_exec();

    // Post variables
     
    '&back_url=http://buddymarks.com/view_group.php?id=938139';
     .= 
    '&form_buddymark=true';
     .= 
    '&form_public_link=true';
     .= 
    '&form_group_type=existing';
     .= 
    '&form_group_id=938139';
     .= 
    '&form_group_title=';
     .= 
    '&form_group_parent_id=0';
     .= 
    '&form_cat_id=2';
     .= 
    '&bookmark_title=hello';
     .= 
    '&bookmark_url=http://im-fun.com';
     .= 
    '&form_description=my blog';
     .= 
    '&form_tags=tags';
     .= 
    '&form_private=2';

    // Curl setup
    curl_setopt(, CURLOPT_URL'http://www.buddymarks.com/add_bookmark.php?action=insert_bookmark');
    curl_setopt(, CURLOPT_POST1);
    curl_setopt(, CURLOPT_POSTFIELDS, );
    curl_setopt(, CURLOPT_HEADER0);
    curl_setopt(, CURLOPT_FOLLOWLOCATION1);
    curl_setopt(, CURLOPT_SSL_VERIFYPEERfalse);
    curl_setopt(, CURLOPT_COOKIEJAR"my_cookies.txt");
    curl_setopt(, CURLOPT_COOKIEFILE"my_cookies.txt");
    curl_setopt(, CURLOPT_RETURNTRANSFER1);
    curl_setopt(, CURLOPT_USERAGENT"Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (.NET CLR 3.5.30729) Creative ZENcast v2.00.14");

    // Insert bookmark
     
    curl_exec();

    // Display the follow location response
    echo ; 
    But unfortunately I was only able to login and couldn't add the bookmark.

    I'm not really sure how to proceed with your example. Does that code just grab the header info and save it as a cookie? Then I'd still need to initiate a LOGIN using the same header info?

    I tried putting the "meat & potatoes" of my code after your code and it seemed to get me further. Instead of the ADD_BOOKMARK.php page redirecting me back to itself after the curl_exec(), it actually says "Public Bookmark not saved!" this time. But I'm not sure how to debug that particular message.
    {{ DiscussionBoard.errors[993962].message }}
  • Profile picture of the author khtm
    Wait a sec!

    It actually did work! Even though it says "Public Bookmark not saved" I checked and it was actually there.

    Sweeeeet

    You're a genius, Max.
    {{ DiscussionBoard.errors[993967].message }}
  • Profile picture of the author maxleadford
    khtm,

    Glad to hear it helped.

    I wrote this as part of a data scraper three years ago to grab some data from a public records website for some real estate stuff. I'm actually a bit foggy on the specifics, but I'm pretty sure that as long as you stay in the same cURL session (in my example $ch), your cookies should stay in place.

    Good luck on your endeavor!
    {{ DiscussionBoard.errors[994118].message }}
  • Profile picture of the author maxleadford
    khtm,

    It just occurred to me that I should have recommended Firebug for Firefox. I've you've not used this plugin, it'll greatly assist you in picking apart your form calls on your target websites.

    You want to use the "net" panel when you send a form submission. It'll list out every URL hit in the submission process and show all the headers and responses from every URL. Insanely useful when there's more than a single URL involved in a submission process. [Those often go overlooked if they load quickly during your submission process and you're not running Firebug or some other logger on port 80 traffic.]
    {{ DiscussionBoard.errors[994268].message }}
    • Profile picture of the author khtm
      Thanks Max! I wasn't aware of that plugin - I'm gonna check it out now.
      {{ DiscussionBoard.errors[994451].message }}

Trending Topics