Hypothetical Possibilities

by 9 replies
10
Let's say that I wanted to create a program that could analyze the front page of Google for a keyword and send back to me that PR of the Top 10 results, how many ads are on top of the page, and how many ads are in the sidebar.

Would this be incredibly difficult to do?

-- j
#programming #hypothetical #possibilities
  • What's difficult here is that google very much dislikes being scraped. So if you are planning to get many pages for many keywords in a short period of time, you'll have to somehow avoid catching captchas all the time - use proxies, dynamic IPs, Tor, whatever.
  • Banned
    [DELETED]
  • [DELETED]
    • [1] reply
    • Which one? Don't think there is an official google API for search results or ads on results pages now... The closest thing is custom search API, but not sure it's what you wanted...
  • You can use Google search API, but Google search API dont show advertisement and they are not free. But again, for checking PR you need to call google toolbar API and google will start showing captchas or blocking your request.
    • [1] reply
    • What exactly is google search API? Where it is?
      • [1] reply
  • Very true. We have made a script which is checking Google for various keywords and parse the results. But as it was pointed out, making it work in the long run is not very simple, because Google doesn't like to be scraped. So what we do is that we are faking different IP addresses, and user agents, etc... and then everything is fine.

    Don't know the answer to the PR question. Most of the APIs which returned Google PR are dead and not working any longer. Is PR really important any longer btw.?
  • Couldn't agree more with you Jay. PR is no longer important... Is this what you are trying to prove with this script?
  • Well it is either the custom search or the old one which is deprecated but still working:
    developers.google.com/web-search/docs/reference

    As far as I recall the results also change if they get queries from different servers
Join the discussion

Next Topics on Trending Feed

  • 10

    Hypothetical Possibilities

    Let's say that I wanted to create a program that could analyze the front page of Google for a keyword and send back to me that PR of the Top 10 results, how many ads are on top of the page, and how many ads are in the sidebar. Would this be incredibly difficult to do?