23 {{ upvoteCount | shortNum }}
23 {{ upvoteCount | shortNum }}

Is there any web-page copying command in PHP?

by truckload-of-thoughts
20 replies
What is that command in PHP that will allow me to copy
an html code of a web page by its URL?
#command #copying #php #webpage
  • Profile picture of the author patfl
    CURL allows that. Check here:

    Grab HTML content with CURL | drupal.org
    {{ DiscussionBoard.errors[293108].message }}
    • Profile picture of the author truckload-of-thoughts
      Hello techymarketer!

      Thank you for that link. I am just surprized why curl is not
      mentioned in any php tutorial that I have read.
      {{ DiscussionBoard.errors[294718].message }}
      • Profile picture of the author patfl
        Originally Posted by truckload-of-thoughts View Post

        Hello techymarketer!

        Thank you for that link. I am just surprized why curl is not
        mentioned in any php tutorial that I have read.
        Yes, that's what makes it so difficult to learn well, but once you know how it works, man, sky's the limit!
        {{ DiscussionBoard.errors[302622].message }}
  • Profile picture of the author xga
    If fopen wrappers is enabled in your php, you can the file_get_contents() function to get the html code.

    For example,

    $str = file_get_contents("hxxp://somedomain.com");
    echo $str;
    {{ DiscussionBoard.errors[293305].message }}
    • Profile picture of the author truckload-of-thoughts
      xga,
      can you, please, tell me what is wrappers and how I can enable it?
      {{ DiscussionBoard.errors[294721].message }}
      • Profile picture of the author xga
        Originally Posted by truckload-of-thoughts View Post

        xga,
        can you, please, tell me what is wrappers and how I can enable it?
        Sorry for my late response. I think Scott did a excellent job explaining the allow_url_fopen.

        If you are on a shared hosting account, you might want to consider going for a VPS. Shared hosting account are too restrictive for any serious PHP programming.
        {{ DiscussionBoard.errors[296059].message }}
        • Profile picture of the author truckload-of-thoughts
          Originally Posted by xga View Post

          If you are on a shared hosting account, you might want to consider going for a VPS...
          Thank you. (I didn't even know about the existence of VPS before I read your post).
          {{ DiscussionBoard.errors[296221].message }}
  • Profile picture of the author Scott Carpenter
    Create a php file that has this in it:

    PHP Code:
    <?php
    phpinfo    ();
    ?>
    Open that up in your browser and you should see a bunch of information about PHP. Search for the following text: allow_url_fopen - if the "Local Value" column says "On", then you can use the following methods to get the code of a page: (this isn't an exhaustive list, just a few examples)

    Method 1: (easiest)


    <?php
    $html = file_get_contents('http://url.here/');
    ?>


    Method 2:


    <?php
    $f = fopen('http://url.here/', 'r');
    $html = '';
    while ($line = fread($f, 1024))
    {
    $html .= $line;
    }
    fclose();
    ?>


    Method 3:


    <?php
    $html = file('http://url.here');
    ?>


    Note about method 3: $html will be an array with each element of the array being one line in the code. For example:


    $html = array(
    [0] => "<html>",
    [1] => "<head>",
    [2] => "<title>Website title here</title>",
    and so on
    );
    {{ DiscussionBoard.errors[294868].message }}
  • Profile picture of the author Scott Carpenter
    OK seriously, we can't show a dollar sign in code tags?
    {{ DiscussionBoard.errors[294873].message }}
    • Profile picture of the author truckload-of-thoughts
      Originally Posted by Scott Carpenter View Post

      ...we can't show a dollar sign in code tags?
      HA-HA-HA!!!

      Hello, Scott!!!

      Thank you for all these methods you've presented.

      Originally Posted by Scott Carpenter View Post

      ... Search for the following text: allow_url_fopen - if the "Local Value" column says "On", then you can use the following methods
      but what if it says "off", is there any way to change the
      configuration or it's firmly set-and-stone and solely on the
      discretion of the server?
      {{ DiscussionBoard.errors[295055].message }}
      • Profile picture of the author floatingatoll
        Originally Posted by truckload-of-thoughts View Post

        but what if it says "off", is there any way to change the
        configuration or it's firmly set-and-stone and solely on the
        discretion of the server?
        If it says "off", then you can turn it on by adding an entry to your server's php.ini configuration file enabling "allow_url_fopen". This is the only way to enable it when it's disabled.
        {{ DiscussionBoard.errors[295241].message }}
        • Profile picture of the author Scott Carpenter
          Originally Posted by floatingatoll View Post

          If it says "off", then you can turn it on by adding an entry to your server's php.ini configuration file enabling "allow_url_fopen". This is the only way to enable it when it's disabled.
          Bear in mind - if you are on a hosted account, you likely don't have access to the server's php.ini file. Some hosts allow you to put a php.ini in your webserver's root directory, with values that override the server's php.ini values. that's what I would try first, in fact.

          Create a file called php.ini and put it in your root directory. In that file, put the following line:

          Code:
          allow_url_fopen = On
          Then, access your script that has the call to phpinfo() again and see if the value says "On". If it does, you're in luck!
          {{ DiscussionBoard.errors[295289].message }}
          • Profile picture of the author truckload-of-thoughts
            Originally Posted by Scott Carpenter View Post

            ...Then, access your script that has the call to phpinfo() again and see if the value says "On". If it does, you're in luck!

            Scott, thank you very much for all your explanations and
            for your time. I'll be sure to follow up on all your instructions
            given here as soon as my another more fundamental
            PHP-related problem, which I am discussing in "Basic PHP
            help is needed" thread, is solved.
            {{ DiscussionBoard.errors[296214].message }}
          • Profile picture of the author truckload-of-thoughts
            Originally Posted by Scott Carpenter View Post

            Bear in mind - if you are on a hosted account, you likely don't have access to the server's php.ini file. Some hosts allow you to put a php.ini in your webserver's root directory, with values that override the server's php.ini values. that's what I would try first, in fact.

            ...then access your script that has the call to phpinfo() again and see if the value says "On". If it does, you're in luck!
            Alas, it still says "no"

            I think I will start a new thread asking people if they know any free PHP-supporting free host with allow_url_fopen enabled
            {{ DiscussionBoard.errors[300090].message }}
        • Profile picture of the author truckload-of-thoughts
          Originally Posted by floatingatoll View Post

          ...This is the only way to enable it when it's disabled.
          Thank you very much.
          {{ DiscussionBoard.errors[296209].message }}
  • Profile picture of the author Darren Mothersele
    Excellent explaination of file_get_contents etc.

    Once you've got the HTML code you probably want to do something with it. In this case take a look at Simple HTML DOM -

    PHP Simple HTML DOM Parser

    With this awesome library you can parse and extract parts of the code, and deal with the HTML code from the page as a PHP object. Very useful!
    {{ DiscussionBoard.errors[350795].message }}
    • Profile picture of the author truckload-of-thoughts
      Thanks, but, before parsing any HTML code, I need to know what
      kind of script I could run on a free remote server, so that that script
      would periodically open the desired web pages and store their HTML
      code somewhere on the server or at least mail it to my mail box.

      Believe it or not, but, after asking so many questions on forums, I
      still haven't figured out how to do it. It seems to me that there are
      only three possible ways:

      1) PHP;
      2) ASP;
      3) C++;

      In PHP, as you can see in this thread, I stumbled on the problem
      of not being able to even open any URL because on free PHP servers
      the "allow_url_fopen" value is not enabled (well, I was also advised
      here to use curl with PHP, but I haven't tried it yet; I am afraid curl
      is good for performing processes on your own computer, but not on
      any free remote server);

      In ASP I was able to go further - I was able to get a script that
      would open URLs and display pages, but alas, I can't automatically
      create or even open any files there, as free ASP servers, being afraid
      of viruses, disable those functions, too. Right now I am working on
      how to get the opened HTML code emailed to me in ASP.

      As for C++ way, I haven't started to learn it yet.
      {{ DiscussionBoard.errors[353074].message }}
  • Profile picture of the author improvingtheweb
    For security purposes, it's best that you use curl and set allow_url_fopen to off in your php.ini
    {{ DiscussionBoard.errors[377565].message }}
  • Profile picture of the author rarebiz
    I've search on google, and found some tutorial using SnoopyPHP.
    Hope it can help you.

    Rainbow Hat SEO PHP4: How to Steal from Yahoo! (another Web Screen Scraper)
    Rainbow Hat SEO PHP5: Screen scraping with DOM and XPath
    {{ DiscussionBoard.errors[378286].message }}

Trending Topics