19 {{ upvoteCount | shortNum }}
19 {{ upvoteCount | shortNum }}

HELP- Scraper? Spider? Bot? What Do I NEED?

by griffin82
16 replies
Sorry in advance for my lack of knowledge on this topic but I'm hopeful someone can help point me in the right direction.

I'm hoping someone will know of a tool or service that can help me scan the web looking for websites using a specific div id that is common on all the sites using this texting service.

My goal is to return a list of car dealers using Edmonds Car Code on their website. They use a click to chat or text widget that sits on the side or bottom of the page and opens a pop up window to initiate the chat/text session.

Thanks in advance!
#bot #scraper #spider
  • Profile picture of the author Joe Ray
    First, you should get a list of URLs for all the car dealers. Then, someone has to write a code for you to create a bot which will crawl these URLs and return a list with the sites that are using the div you're looking for.
    Signature

    {{ DiscussionBoard.errors[10365671].message }}
    • Profile picture of the author griffin82
      I guess that's the problem, I don't have a list of URL's.

      Is there anything else that could be used from the source code that could be used, like the link address that opens the modal window?
      {{ DiscussionBoard.errors[10365741].message }}
  • Profile picture of the author Joe Ray
    Finding the div is not the problem, that's pretty easy if you have a list of URLs. The problem is to crawl the entire web to find a div, that's not so easy.
    Signature

    {{ DiscussionBoard.errors[10365743].message }}
  • Profile picture of the author Joe Ray
    I googled it and there are some car dealer lists available online that include their websites. I didn't really look to hard but I found one list for all the used car dealers, I am sure you can find the same type of lists for new cars.
    Signature

    {{ DiscussionBoard.errors[10365749].message }}
    • Profile picture of the author griffin82
      Ok. Thanks. After getting a list, how would I proceed?

      Would this require a custom build? What would I request, a bot to search these specific sites for a div?

      Would you look on fiver or elsewhere?
      {{ DiscussionBoard.errors[10366336].message }}
  • Profile picture of the author Joe Ray
    Yes, you need a custom built bot, actually 2 bots. One bot to extract all the links from the website that contains the URLs for the car dealers and the other bot is to crawl the extracted URLs and return a list of URLs that contain the div you are looking for.

    Fiver won't work for this. You can't get this done for $5. This will probably cost you a couple of hundred dollars.

    To scrape the links from the car dealer sites, you may have to get their permission if their ToS doesn't allow scraping. In any case, that will be your decision as this is kind of a gray area.

    To find the div on the sites should be ok because the bot will not extract anything from those sites, it will only crawl the URLs. Search engines routinely do that every day.

    PS: search engines also extract data from websites routinely and that doesn't seem to bother anyone...
    Signature

    {{ DiscussionBoard.errors[10366836].message }}
  • Profile picture of the author yukon
    Banned
    My advice is try & do the easiest stuff first.

    Find a dealer site where you know they have the Edmonds Car Code on their website. Look at the Google cache (text version) to see If there's a footprint (anchor-text, etc...). Next, search that footprint on google SERPs.

    Post a link to a known car dealership site with the Edmonds code, I'll check the cache for a footprint.
    {{ DiscussionBoard.errors[10366843].message }}

Trending Topics