Crawling all css files on the internet

by 9 replies
10
Hi,

I have come up with an business idea and the only way to implement it cost effectively would be to crawl sites' css files to find certain information, then return a list of sites containing this information. So is there a search engine that could do this?

Cheers Don
#programming #crawling #css #files #internet
  • Don't know of any search engines that crawl CSS files, but you could create a scraper that crawls specified websites and returns the info from each site's CSS files, then parse that info however you want.

    Although if you want to crawl ALL of the CSS files on the internet, you'd probably need at least a couple hundred servers, an enterprise-level ISP with a really fast connection, and about 10 years of time. :p
    • [2] replies
    • Hi,

      I've just made an offer to google for few hundred gillion so I can implement the idea. I've actually got Larry on hold while I'm writing this

      There is a site where you can search all the html files on the net, for a very cheap price , so if you can do that I can't see why css files would be anymore difficult. I suppose there is just no demand for a css search engine.

      As for searching a selected subset of sites that's a possibility it would probably cut it down to a few million. But the truth is I was hoping there would be an "off the shelf" solution as it isn't a big business opportunity. A site, based on the information might ,at best, generate 500-1000 dollars a week in revenue.

      I came up with the idea as it will solve a personal problem that has cost me a hundred fifty bucks and probably a hundred frustrating hours of my time. I also know there are thousands of other people out there just like me. A site, based on the information scraped from the css files, would solve our problem

      Cheers Don
      • [1] reply
    • And a garage. Every great IT company started in a garage.
  • Google. Use their api, or crawl it the old way by regex'ing search results. They have file filters and all available.
  • We believe Google is already indexing CSS files and they use some portion of them to understand what color is the background or the contents in the page. But they just don't show results from them, since users want to see English and not Codes.
  • This works fairly well.

    Add this to your google search query:

    Code:
    filetype:css
    Source, which is coincidentaly the first google result for 'google search through css files':
    How to search CSS / JavaScript files with Google? - Web Applications Stack Exchange

    Although, if you can't type your questions into google I am unsure of how much success you will have with more complicated functions such as this though.
Join the discussion

Next Topics on Trending Feed

  • 10

    Crawling all css files on the internet

    Hi, I have come up with an business idea and the only way to implement it cost effectively would be to crawl sites' css files to find certain information, then return a list of sites containing this information. So is there a search engine that could do this?