10 {{ upvoteCount | shortNum }}
10 {{ upvoteCount | shortNum }}

Hypothetical Possibilities

by TheArticlePros
9 replies
Let's say that I wanted to create a program that could analyze the front page of Google for a keyword and send back to me that PR of the Top 10 results, how many ads are on top of the page, and how many ads are in the sidebar.

Would this be incredibly difficult to do?

-- j
#hypothetical #possibilities
  • Profile picture of the author phpg
    What's difficult here is that google very much dislikes being scraped. So if you are planning to get many pages for many keywords in a short period of time, you'll have to somehow avoid catching captchas all the time - use proxies, dynamic IPs, Tor, whatever.
    {{ DiscussionBoard.errors[7232040].message }}
  • Profile picture of the author TheArticlePros
    [DELETED]
    {{ DiscussionBoard.errors[7232146].message }}
    • Profile picture of the author phpg
      Originally Posted by JaRyCu View Post

      What if I just used the Google API and made it 100% legit?

      -- j
      Which one? Don't think there is an official google API for search results or ads on results pages now... The closest thing is custom search API, but not sure it's what you wanted...
      {{ DiscussionBoard.errors[7232230].message }}
  • Profile picture of the author Oswald Joshua
    You can use Google search API, but Google search API dont show advertisement and they are not free. But again, for checking PR you need to call google toolbar API and google will start showing captchas or blocking your request.
    {{ DiscussionBoard.errors[7247184].message }}
    • Profile picture of the author phpg
      Originally Posted by Oswald Joshua View Post

      You can use Google search API, but Google search API dont show advertisement and they are not free. But again, for checking PR you need to call google toolbar API and google will start showing captchas or blocking your request.
      What exactly is google search API? Where it is?
      {{ DiscussionBoard.errors[7247267].message }}
      • Profile picture of the author cgimaster
        Originally Posted by phpg View Post

        What exactly is google search API? Where it is?
        developers.google.com/custom-search/docs/dev_guide
        {{ DiscussionBoard.errors[7247962].message }}
        • Profile picture of the author phpg
          Originally Posted by cgimaster View Post

          developers.google.com/custom-search/docs/dev_guide
          Well, i've mentioned custom search earlier myself in this thread, but it's different, results you get there are not the same you get when just searching, and you can't really use it to determine sites' places in real results etc.
          {{ DiscussionBoard.errors[7247986].message }}
  • Profile picture of the author WebThinker
    Very true. We have made a script which is checking Google for various keywords and parse the results. But as it was pointed out, making it work in the long run is not very simple, because Google doesn't like to be scraped. So what we do is that we are faking different IP addresses, and user agents, etc... and then everything is fine.

    Don't know the answer to the PR question. Most of the APIs which returned Google PR are dead and not working any longer. Is PR really important any longer btw.?
    Signature
    {{ DiscussionBoard.errors[7247220].message }}
  • Profile picture of the author WebThinker
    Couldn't agree more with you Jay. PR is no longer important... Is this what you are trying to prove with this script?
    Signature
    {{ DiscussionBoard.errors[7247270].message }}
  • Profile picture of the author cgimaster
    Well it is either the custom search or the old one which is deprecated but still working:
    developers.google.com/web-search/docs/reference

    As far as I recall the results also change if they get queries from different servers
    {{ DiscussionBoard.errors[7247998].message }}

Trending Topics