7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Amazon, Newegg and Tigerdirect script

by zaco
5 replies
Hey guys!,

I was wondering if it is possible to create a script that runs on my dedicated server to collect the products from those 3 sites and post them on my site ( that is the basic idea).. the first concern is would they allow me to scrape that data? or they will block / ban my ips? I will be collecting almost all of their products which will take a while.. like months since I want to do it slowly.

I know Amazon has an API but would it allow me to scrape that much Data? I did email them but they never got back to me to tell if it is okay.. any help would be appreciated.
#amazon #newegg #script #tigerdirect
  • Profile picture of the author dtang4
    It is probably against their terms for you to do this.

    Therefore, you will need to use your own scraper (and not their API). Then, you will need to have pauses in your script, maybe change up IPs, etc., to mimic human behavior. Otherwise, if you run a script that just persistently scrapes, the server will ban you after a couple hours (probably).
    Signature
    Get Your AI Audit
    {{ DiscussionBoard.errors[4741539].message }}
    • Profile picture of the author zaco
      Originally Posted by dtang4 View Post

      It is probably against their terms for you to do this.

      Therefore, you will need to use your own scraper (and not their API). Then, you will need to have pauses in your script, maybe change up IPs, etc., to mimic human behavior. Otherwise, if you run a script that just persistently scrapes, the server will ban you after a couple hours (probably).

      Do you know any applications that I can use to test if they ban my ip?
      {{ DiscussionBoard.errors[4741591].message }}
  • Profile picture of the author KirkMcD
    Let's just cut to the chase, why the hell would you want to do something so foolish?
    By the time you are finished, everything from the beginning would be out of date and you would need to start over again.
    Heck, Amazon says not to display anything pulled from the API after an hour because it may no longer be valid.
    {{ DiscussionBoard.errors[4749821].message }}
    • Profile picture of the author zaco
      Well what I am trying to do is collect the product lists and not prices so they can change whatever they want.. The problem is I cannot find if newegg / tigerdirect provide a datafeed.. does anybody know? if yes then I don't need to scrape it... since it is available in teh datafeed
      {{ DiscussionBoard.errors[4750362].message }}
  • Profile picture of the author Ken Durham
    I think as a reseller for Tiger you have access to feeds.
    Using any of the data from any of the sites without permission is illegal, but you already know that.

    A lot of that traffic that some people think they get, are actually scrapers.
    I don't agree with what you are doing, but many have given it a try and learn from it. Mostly what not to do. You'll figure things out.

    You can always DIY on the scraper. You'll need the skills if you are going to the dark side
    » CURL Page Scraping Script - Digerati Marketing
    Signature

    yes, I am....

    {{ DiscussionBoard.errors[4753769].message }}

Trending Topics