How to crawl a website & build a product feed?

Profile picture of the author willyboy104 by
I've generally outsourced this activity for most of my clients however I am researching in to building product feeds for use with Google Shopping myself however I am wondering how do I go about crawling a site to get all the appropriate data, such as:

Product URL
Image URL
Product Description


I could obviously do this manually, but when a client has in excess of 5,000 products it is not feasible.

Anyone got any advice?

Thank you!
#seo #build #crawl #feed #product #website

  • Profile picture of the author willyboy104
    Thanks, but no thanks - for the pointless, blatant promotion of your own services/lead generation in your site links.

  • Profile picture of the author Sillysoft
    I built something like this a long time ago. It was a php spider that would spider a site, get the product information and put it into an xml file then auto submit to places like froogle.

    What site are you trying to obtain these products? I wouldnt mind seeing if I could get my old script to try and grab those products.

Next Topics on Trending Feed

  • 1 {{ upvoteCount | shortNum }} 5

    The day is great, itís warm outside, fresh wind is blowing, it was raining yesterday, though. Iím feeling a bit tired after 7 days in Costa Rica, where if itís not raining then itís going to rain )) All of this inspire me greatly to finally get back to business and share the info with forum members. I've been a freelancer for many years and these days I try to join the forum discussions, where I can get useful tips on issues that interest me, because Iím not only used to get info, but also share with the interested users. Iíve made lots of videos and pictures, and it means that Iíll make a short instruction course on how to make a video on YouTube. I think almost all SEO-optimizers have to face it every single day, the sheer volume of projects from freelancers is incredible as well as the questions on YouTube promotion forum.

  • 2 {{ upvoteCount | shortNum }} 5

    Hi. I won't ask the same question over and over again "How to make money online". All i will simply ask is, to recommend me some books to get so i can answer that question for myself. I need to educate myself, not go around in circles asking the same old question that gets me nowhere.

  • 0 {{ upvoteCount | shortNum }} 5

    Over the past two months, I have been working on an interesting project at my company which is supposed to be a very big one. So at first, as a team leader and core maker, I spent quite a lot of time to find a good starting point, I meant stuff like how to structure the project, which frontend stack to use, how that stack connects to server, how to set up and bring all those things together, etc, to satisfy all the projectís requirements. Finally, I settled down with Meteor for backend, React for frontend, and Vanilajs ES6 MVC for the app extension (this was the most interesting part of the project. I will talk about it in another blog). At first things went well and smooth. Needless to say that React was so amazing that I completely changed my thoughts when programming.