RESOLVED-Accessing site via a proxy server with PHP file_get_contents

12 replies
ALL NOW SORTED THANKS KIRKMCD & WAYFARER
Hi Guys,
Looking for an answer to this isse that's had me going around in circles for 2 days and now I'm at a brick wall. There are number of resources that purport to have the solution but they all look like they copied the same script so i am now not convinced it is correct.

Hereis the short test script I have used to test this but I get the following error.
Warning: file_get_contents(The AA: Breakdown Cover, Car Insurance, Home Insurance, Loans, Motoring Advice, Travel and Maps) [function.file-get-contents]: failed to open stream: Connection timed out in /home5/XXXXXXXX/public_html/mkm/tools/test.php on line 11


The script is:
_____________

<?php

function file_get_contents_proxy($url,$proxy){

// Create context stream
$context_array = array('http'=>array('proxy'=>$proxy,'request_fullu ri'=>true));
$context = stream_context_create($context_array);

// Use context stream with file_get_contents
$data = file_get_contents($url,false,$context);

// Return data via proxy
return $data;

}



$url = "http://www.theaa.com";
$proxy = "tcp://132.254.101.52:3128";

file_get_contents_proxy($url,$proxy);
Echo $data

?>
________

$url is an arbitary wesite
$proxy is a randomly chosen public proxy that I have tested both with an online checker and by adding it to my browser config.

If I just use the file_get_contents I can access the file directly but I cannot access it via the proxy.

Anyone familiar with accessing sites/site files via a proxy server?
Really appreciate your help

Will definitely find some kind of "reward" for anyone who gives me the solution.
Or tell me how much

Thanks
Tony
#accessing #filegetcontents #php #proxy #server #site
  • Profile picture of the author KirkMcD
    Try setting proxy to
    $proxy = "132.254.101.52:3128";
    instead of
    $proxy = "tcp://132.254.101.52:3128";

    I think I read somewhere people having problems with that.
    It could also be that the function is blocked for external urls.

    With all that being said, I usually use CURL for reading external web pages.
    {{ DiscussionBoard.errors[3812289].message }}
    • Profile picture of the author Tony Marriott
      Many thanks Kirk,

      Unfortunately that has made no difference.
      The main script that I created has been working fine on two differnt servers/hostings before I tried to add the proxy (and still does), So the file_get_content is definetly enabled.

      I haven't used curl as I eventually want to distribute the script and I understand that it isn't always enabled on servers.

      But if I can't resolve this then curl may be a good option. Do have know of any resources that would help me sort this in curl or id there something you feel might be a possible solution?

      many thanks
      Tony
      {{ DiscussionBoard.errors[3812596].message }}
      • Profile picture of the author KirkMcD
        Originally Posted by Tony Marriott View Post

        I haven't used curl as I eventually want to distribute the script and I understand that it isn't always enabled on servers.
        You'll run into the same problem with fopen, but a host may be more likely to enable curl, than fopen.

        Here is a curl fetch:
        $user_agent is a user agent string
        $proxy is the proxy ip with port
        $postdata is used when POSTing data to a url.

        function fetch($url, $postdata = false) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        if ($user_agent !== '') {
        curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
        }
        curl_setopt($ch, CURLOPT_HTTPHEADER, array("Expect:"));
        if (!ini_get('open_basedir') && !ini_get('safe_mode')) {
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
        }
        if ($postdata) {
        curl_setopt($ch, CURLOPT_POST, 1);
        curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
        }
        if ($proxy !== '') {
        curl_setopt(, CURLOPT_PROXY,$proxy );
        }
        $response = curl_exec($ch);
        curl_close($ch);
        return $response;
        }
        {{ DiscussionBoard.errors[3816303].message }}
        • Profile picture of the author Tony Marriott
          WOW!
          Thanks Kirk
          I take your point about curl. I was probably fooling myself to aviod looking into someting that's not in my comfort zone. looks lie it's time to step in!

          Script looks good I'll go through it tomorrow with a clear head.

          Many thanks

          I'll let you know how it goes.

          cheers
          Tony
          {{ DiscussionBoard.errors[3818525].message }}
          • Profile picture of the author spamfighter
            +1 for CURL

            many hosting providers switch off: allow_url_fopen
            for security reasons.

            Most providers have CURL installed or will install it for you if you ask them.
            {{ DiscussionBoard.errors[3829838].message }}
            • Profile picture of the author Tony Marriott
              Originally Posted by spamfighter View Post

              +1 for CURL

              many hosting providers switch off: allow_url_fopen
              for security reasons.

              Most providers have CURL installed or will install it for you if you ask them.
              +1 for CURL but unfortunately it has not resolved my primary issue.

              Or at least I haven't (may not be CURL's fault!!)
              {{ DiscussionBoard.errors[3857075].message }}
        • Profile picture of the author Tony Marriott
          Originally Posted by KirkMcD View Post

          You'll run into the same problem with fopen, but a host may be more likely to enable curl, than fopen.

          Here is a curl fetch:
          is a user agent string
          is the proxy ip with port
          is used when POSTing data to a url.

          function fetch(, = false) {
          = curl_init();
          curl_setopt(, CURLOPT_URL, );
          curl_setopt(, CURLOPT_RETURNTRANSFER, true);
          if ( !== '') {
          curl_setopt(, CURLOPT_USERAGENT, );
          }
          curl_setopt(, CURLOPT_HTTPHEADER, array("Expect:"));
          if (!ini_get('open_basedir') && !ini_get('safe_mode')) {
          curl_setopt(, CURLOPT_FOLLOWLOCATION, 1);
          curl_setopt(, CURLOPT_MAXREDIRS, 10);
          }
          if () {
          curl_setopt(, CURLOPT_POST, 1);
          curl_setopt(, CURLOPT_POSTFIELDS, );
          }
          if ( !== '') {
          curl_setopt(, CURLOPT_PROXY, );
          }
          = curl_exec();
          curl_close();
          return ;
          }

          Hi Kirk,

          Thanks again for your detailed script. I've now had time to go through it and test. Unfortunately I am still unable to get the proxy working. I can open a file (URL) no problem but as soon as I add the proxy it just does nothing.

          I have simplified the script to it's bare bones and have the following
          +++++++++++++++++++++++++++

          $filename is an arbitary web site

          $proxy is a free public proxy but was Working at the time of posting


          $filename = "http://www.Google.com";

          $proxy = "60.175.203.243:8080";

          $ch = curl_init();
          curl_setopt($ch, CURLOPT_URL, $filename);
          curl_setopt($ch, CURLOPT_HEADER, 1);
          curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
          curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
          curl_setopt($ch, CURLOPT_PROXY, $proxy);
          $data = curl_exec($ch);
          curl_close($ch);
          echo $data;
          ?>
          ++++++++++++

          Put nothing in the proxy string and it all works. Add $proxy and it does not.
          Have tried qualifying proxy with FTP and HTTP but still the same.

          Still very happy to get an answer here and very happy to pay for a solution

          cheers
          Tony
          {{ DiscussionBoard.errors[3857096].message }}
  • Profile picture of the author wayfarer
    If it was working on other servers, it just may be the current server does not have the allow_url_fopen option enabled in php.ini ... check that.
    Signature
    I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
    {{ DiscussionBoard.errors[3812712].message }}
    • Profile picture of the author Tony Marriott
      Hi Wayfarer,

      To clarify. I have tried on two servers and both have exactly the same results.
      opening the url with either fopen or file-get_contents works fine.

      It only fails to open when I add in the stream_context_create to try and open it via the proxy server.

      cheers
      Tony
      {{ DiscussionBoard.errors[3812919].message }}
  • Profile picture of the author wayfarer
    There's an alternative solution that you may not have considered, one that I've used to scrape public information with: an HTTP proxy. An HTTP proxy is free to set up, and can be done with Google's AppEngine. I own 5 proxies that I modified for my purposes, which came from a free Python script. This is useful if all you need to do is read content from external websites, since everything is done with plain HTTP requests (file_get_contents or cURL requests, without using the proxy argument).

    The proxy script I used was this: mirrorrr - Mirrorrr - Web Cache - Google Project Hosting ... If you need help, PM me. Or, we could work out a deal where you use my proxies. They're super fast and reliable, since they're on Google's cloud-servers.
    Signature
    I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
    {{ DiscussionBoard.errors[3858032].message }}
    • Profile picture of the author Tony Marriott
      Hi Wayfarer,

      Many thanks for the tip and the offer.

      This script is part of a larger script that will become a distributed product so unfortunately I would need a set of proxies for every copy which would not be viable.

      It looks an interesting idea though and I can see a lot of advantages of having that setup.

      I'll keep the idea in mind as I am sure it will be resolution to something else I have in mind.

      I have to get this project finished first but I may well take you up on your offer of help later.

      thanks
      Tony
      {{ DiscussionBoard.errors[3859873].message }}
      • Profile picture of the author Tony Marriott
        Hi Guys,

        Just an update to say I evetually got teh problem resolved.

        It looks like my main issue wasn't the script itself but the proxy ports. Or more accurately the lack of holes for them in the hosting firewall.

        Anyway I have now started looking at CURL, so thats new for me.

        Many thanks to KirkMcD and Wayfarer for all your help. I have PM'd you both a small thank you.
        {{ DiscussionBoard.errors[3901077].message }}

Trending Topics