12 {{ upvoteCount | shortNum }}
12 {{ upvoteCount | shortNum }}

legality of search engines

by Calvin Shoop
11 replies
  • SEO
    • |
I'm working on building a meta-search engine that aggregates listings from five different classifieds sites.

Do I need permission to run crawlers on these sites, or can I just do it? What is the difference between what I want to do and what Google does?
#engines #legality #search
  • Profile picture of the author OptedIn
    Originally Posted by Calvin Shoop View Post

    What is the difference between what I want to do and what Google does?
    Google has an army of world-class coders and billions of dollars to fund any project.

    You asked!
    Signature

    "He not busy being born, is busy dying." - Bob Dylan • "I vibe with the light-dark point. Heavy." - Words that Bob Dylan wishes he had written.

    {{ DiscussionBoard.errors[11004303].message }}
  • Profile picture of the author Walt SEO
    Originally Posted by Calvin Shoop View Post


    Do I need permission to run crawlers on these sites
    Public information.
    {{ DiscussionBoard.errors[11004399].message }}
    • Profile picture of the author BigFrank
      Banned
      Originally Posted by Walt SEO View Post

      Public information.
      Once they realize you are crawling, you will get blocked. The information may be public, but you don't have a right to scrape it from their site. Try it and you'll see.

      Frank
      {{ DiscussionBoard.errors[11004554].message }}
  • Profile picture of the author Calvin Shoop
    So would it be better to work out a deal with them beforehand and show them how it could be a good thing for them?
    {{ DiscussionBoard.errors[11004582].message }}
    • Profile picture of the author BigFrank
      Banned
      Originally Posted by Calvin Shoop View Post

      So would it be better to work out a deal with them beforehand and show them how it could be a good thing for them?
      Good luck with that. I doubt very much that they share your vision.

      Frank
      {{ DiscussionBoard.errors[11004914].message }}
  • Profile picture of the author Acreaty M
    Google is most preferable search engine in the world.. Google wants every site having quality content good page speed and mobile compatible sites..
    {{ DiscussionBoard.errors[11004602].message }}
  • Profile picture of the author thatkeywordguy
    Originally Posted by Calvin Shoop View Post

    Do I need permission to run crawlers on these sites, or can I just do it?
    In the early days, some websites would get mad at Google. And send them a legal letter telling them to STOP CRAWLING MY WEBSITE!

    Also, if the company puts a NOINDEX tag on their site, then google won't bother the website and hence, doesn't use their website resources.

    So technically, if the people you're crawling tell you to STOP CRAWLING me, then they have a case. Though its iffy.
    Signature

    Get Keyword Reseacher at CleverGizmos.com.

    {{ DiscussionBoard.errors[11006674].message }}
  • Profile picture of the author David Jones42
    you can run cralwer on any sites there is no rules.
    Signature

    Top mobile app, games, and web development company in India AIS Technolabs

    {{ DiscussionBoard.errors[11006908].message }}
    • Profile picture of the author BigFrank
      Banned
      Originally Posted by David Jones42 View Post

      you can run cralwer on any sites there is no rules.
      A now, a message from the uninformed! lol

      Frank
      {{ DiscussionBoard.errors[11007159].message }}
  • Profile picture of the author Sclark
    Most probably if a website owner wouldn`t like the website to be cralwed by any bot - it`ll be specified it in the robots.txt, or any other way. While crawling any website, you may just get detected for excessive queries for instance, and get blocked for a while, or your bot may also appear in their robots file after that. Until then, the info is public indeed.
    {{ DiscussionBoard.errors[11006960].message }}
  • Profile picture of the author yukon
    Banned
    Originally Posted by Calvin Shoop View Post

    I'm working on building a meta-search engine that aggregates listings from five different classifieds sites.

    Do I need permission to run crawlers on these sites, or can I just do it?
    Do you?

    It's 5 sites, read their TOS.





    Originally Posted by Calvin Shoop View Post

    What is the difference between what I want to do and what Google does?
    The difference is you have 5 sites that can reject you while Google has a billion sites to pick from.

    Google also only crawls a dozen or so pages per visit. If you go full throttle scraping a domain it will be the same as a DoS attack which can get your ass in trouble If a webmaster gets pissed off enough day after day hammering their bandwidth and getting their site shut down until they upgrade hosting ($$).
    {{ DiscussionBoard.errors[11007156].message }}

Trending Topics