Would a SERP scraper make a good WSO?

10 replies
  • SEO
  • |
Don't know if this is the right forum to be asking this in but would a PHP script that scrapes Google search result pages and pulls URL's from it make a good WSO?

Mind you this is not a script that violates the Google TOS in that this script operates on SERP result pages that you save locally to your hard disk (there is nothing in the Google TOS saying you can't do that...what they are concerned with is automated access to their search service taking up bandwidth and otherwise disrupting their service).

Basically you would enter a "site:domain.com" (for example) into Google, go the last page, click on the include all results link provided, then go back to page 1 and sequentially copy each page of results to your hard drive (pressing the Next button between each such page).

Then you would run my script. It produces a CSV file that includes the indexed URL's (in alphabetical order), HTTP Status code of trying to access each such link (there usually are just 200 OK Statutes as Google doesn't keep broken links in their index for very long) and shows if any pages are redirected and if so to where.

The CSV can be imported into Google Docs, spreadsheets and what have you.

This is mostly useful for URL cleanup at a site to help detect duplicate URL's, faulty URL's, etc. for SEO purposes but I am wondering if it is generally useful to others here as a potential WSO offering?

A very successful internet marketing friend of mine says that such scripts are all over so the demand for this may not be that great but I wanted to get input from others before abandoning this as a WSO type offering idea.

Oh..my script is only useful for sites that have less than 1000 pages as Google only ever shows 10 pages of results (when set to 100 results per page) no matter how big a site one has.

Any input would be appreciated.

Thanks.

Carlos
#good #make #scraper #serp #wso
  • Profile picture of the author Nikolas
    I've searched this thing lately for a project of my company and there is nothing available, so I guess that your product would have demand.

    But I would prefer something more automated, a solution that a newbie could use.

    Just my 2 cents
    {{ DiscussionBoard.errors[5385764].message }}
    • Profile picture of the author carlos123
      Hmm...interesting. Well, even though what my friend said might be true...if the knowledge of where to find such scripts is not something that is generally known...there might be some opportunity for mine.

      As far as user friendly is concerned...if saving SERP results to individual files in a directory is too difficult I am not sure that I can do anything to make it easier.

      People in general (not referring to you in particular Nikolas) need to start learning how to do the most basic things on their computers like saving a page to their local hard drive, changing settings used by Google during searches, running a script from the command line, and other such things which are not at all complicated.

      Instructions would be very simple as in.

      1. Go to Google
      2. Set it to display 100 results per page (separate instructions for how to do that would be included)
      3. Enter "site:<domain>" into Google where domain is your site
      4. Go to last page of results and click the "repeat the search with omitted results included" link
      5. Go back to first page and save the page through your browser (usually Ctrl+S).
      6. Go to each subsequent page and save each page
      7. Run script.

      There is no way around having to do some manual steps that I know of though I suppose a grease monkey script could be developed to automate some of that. Only thing with that is that one would then have to follow other similar steps of instructions to install the seamonkey script which would defeat the purpose of automating these few steps of instructions (in that such would be replaced by another few steps of instructions to install the seamonkey script ).

      Not sure if I will create any WSO with this but for what it's worth those are my further thoughts on making it user friendly.

      Carlos
      {{ DiscussionBoard.errors[5385939].message }}
  • Profile picture of the author WillR
    Originally Posted by carlos123 View Post

    Don't know if this is the right forum to be asking this in but would a PHP script that scrapes Google search result pages and pulls URL's from it make a good WSO?
    Why would you want to limit it to being just a WSO?

    People seem to get it all wrong these days. A WSO is just another way of selling your product -- you shouldn't be building products just for the sole intention of selling them on the WSO forum. You can do that, but what a waste of your resources. Think BIG.

    As for the idea, build it and you will see.
    {{ DiscussionBoard.errors[5387542].message }}
    • Profile picture of the author carlos123
      Originally Posted by WillR View Post

      Why would you want to limit it to being just a WSO?
      Hmm...well...where else am I going to be able to advertise it to a potentially ready market of hot buyers for a measly $20?

      Think BIG.
      I do think big. My problem is that I don't know how to get from the wee itty bit place I am at to the BIG part LOL.

      So starting at $20 to present a WSO seems like step in the right direction...I think.

      Just saying. I still don't know if this is worth marketing at all. I mean one person saying they might find some value in it if I make it more user friendly does not for a captive market make.

      I'll be using this script on 4 or 5 more sites tomorrow that an SEO client of mine finds real useful. I'll have to pick his brain on why so that I can get some real good benefit selling points.

      Carlos
      {{ DiscussionBoard.errors[5387782].message }}
      • Profile picture of the author tpw
        Originally Posted by carlos123 View Post

        So starting at $20 to present a WSO seems like step in the right direction...I think.

        Your numbers are wrong.

        You must be a War Room member before you can submit a WSO.

        That is a one-time $37 fee. (You are covered there.)

        Then to post a WSO is $40 per WSO. Bumps are also $40.
        Signature
        Bill Platt, Oklahoma USA, PlattPublishing.com
        Publish Coloring Books for Profit (WSOTD 7-30-2015)
        {{ DiscussionBoard.errors[5387787].message }}
        • Profile picture of the author carlos123
          Originally Posted by tpw View Post

          Your numbers are wrong.

          You must be a War Room member before you can submit a WSO.

          That is a one-time $37 fee.

          Then to post a WSO is $40 per WSO. Bumps are also $40.
          Oops. I don't know where I got the $20 from. Must have been from way back or something. I am a War Room member so the $37 is not a consideration.

          $40 is still not too bad. I mean I would think this WSO could sell at least 2...I hope . If I sell at least 2 I wouldn't need to bump it and could on to selling it somewhere else where I could tap into the BIG side of things.

          Maybe a landing page with an Adwords campaign or something. Hmm...maybe I could do that first and test out the market for it a bit better. Kinda hard to explain things in a little bitty Adwords ad though.

          I'll work out the kinks in using it with my client first I think. Polish it up you see. Then we'll see.

          Carlos
          {{ DiscussionBoard.errors[5387818].message }}
          • Profile picture of the author CDarklock
            Originally Posted by carlos123 View Post

            Must have been from way back or something.
            The price change from $20 to $40 only happened just last year, so it's not that far back. Same with the War Room requirement.
            Signature
            "The Golden Town is the Golden Town no longer. They have sold their pillars for brass and their temples for money, they have made coins out of their golden doors. It is become a dark town full of trouble, there is no ease in its streets, beauty has left it and the old songs are gone." - Lord Dunsany, The Messengers
            {{ DiscussionBoard.errors[5388823].message }}
        • Profile picture of the author joshuayip
          Originally Posted by tpw View Post

          Then to post a WSO is $40 per WSO. Bumps are also $40.
          This is the first time in my life to know a bump cost money surprise surprise...

          Is this script available now?
          {{ DiscussionBoard.errors[6634466].message }}
  • Profile picture of the author tpw
    You have to ask yourself who would want a SERP scraper, and whether they would be willing to pay you money for something like that.

    I would value a product like that in the same way that you value webinars.
    Signature
    Bill Platt, Oklahoma USA, PlattPublishing.com
    Publish Coloring Books for Profit (WSOTD 7-30-2015)
    {{ DiscussionBoard.errors[5387664].message }}
    • Profile picture of the author carlos123
      Originally Posted by tpw View Post

      I would value a product like that in the same way that you value webinars.
      Touche!

      That brought quite the smile to me LOL. Truly. You have a way with words my good man (for those that might not know, me and Bill have been verybally sparring on another thread about webinars).

      Carlos
      {{ DiscussionBoard.errors[5387748].message }}

Trending Topics