17 {{ upvoteCount | shortNum }}
17 {{ upvoteCount | shortNum }}

Looking for solution to merge, dedupe, and manage millions of customer contacts

by Nysurance
12 replies
We have numerous databases of millions of prospects for numerous vertices. Looking for a good software or cloud base solution that is cost effective in order to manage, merge, and dedupe all contacts. Most solutions we have found so far either can't handle that many records or are beyond our budget . Would appreciate any insight anyone could provide!
#contacts #crm #customer #database #dedupe #manage #merge #millions #solution
  • Profile picture of the author iAmNameLess
    Split up a csv and do it in Excel. Pretty easy.

    You can also do it with pretty much ANY self hosted CRM. I personally like nutshell for a CRM and they can handle millions of contacts/accounts and automatically remove the duplicates as long as you set it up properly.

    Pretty standard stuff... not sure why you couldn't find a solution that could handle it.
    Signature
    {{ DiscussionBoard.errors[10132411].message }}
    • Profile picture of the author Nysurance
      When using excel it can not handle millions of records at once. We need something that can easily handle about 50 single files of 1 million records each and merge into one database that we can then export.

      We currently have a a very robust enterprise class CRM solution but when it comes to importing this much data it is limited to importing 50,000 records at a time and can dedupe but lacks the ability to merge the data. Looking for something designed to do specifically this that is not that labor intensive

      Most other CRM's I have found similar limitations with.

      I appreciate your response and any further ideas or insight you may have
      {{ DiscussionBoard.errors[10133672].message }}
  • Profile picture of the author Peter Lessard
    The truth is working with huge data sets is labor intensive and time consuming and if you are not good at it you will screw things up. I was in the list business for a very long time dealing with 50 million plus records and initially was doing it myself since I am also a programmer/database developer.

    I quickly lost patience though and what I did was find a few guys, believe it was on Elance at the time that lived and breathed databases. Not hard to find someone that will merge the whole thing and dedupe it all under $100 and they will do it properly without totally screwing up all the data.

    Every time I wanted specific queries run like breaking it all down by niche or state I would agree to a set fee to get it done and be provided download links. Generally in the $25 to $50 range and save me a days work and a computer being tied up.
    Signature
    Ready to generate the next million in sales? The Next Million Agency
    {{ DiscussionBoard.errors[10134074].message }}
  • Profile picture of the author fasteasysuccess
    That many, I agree 100% with peter, had a client a few years back that was working with almost 10 miillion people. An insane amount. Plus, this was all collected from him and jv's so all opted in at one time, so...

    Instead of messing with all that you're going through and excel, etc..definitely go with outsourcing it to get it nice and neat. Not sure if constantly in contact with that list, but if not...

    Then to "qualify" to find out who's active, legit and hyper, send multiple emails to grab groups at a time and put on new "fresh list". Can even segment by topics to pull out as many leads as can. Then only deal with those and maybe down the road, blast that millions list again to catch more but always bring them into the active list.

    This way you save money and spend time with the "quality".
    Signature
    {{ DiscussionBoard.errors[10134122].message }}
    • Profile picture of the author savidge4
      Oracle is the answer here. it will handle pretty much anything you can throw at it. Oracle programmers are not quit dime a dozen, but there are many out there. Many un employed out there to be more accurate.

      You might want to go that avenue get the initial task completed then look into the concept of developing your own custom CRM with Oracle that in the long run would probably save you time and money.
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[10134214].message }}
      • Profile picture of the author jamesfreddyc
        Originally Posted by savidge4 View Post

        Oracle is the answer here. it will handle pretty much anything you can throw at it. Oracle programmers are not quit dime a dozen, but there are many out there. Many un employed out there to be more accurate.
        I work with both Oracle and SQL Server... the SQL to filter as the OP wants is going be the same SQL statements across databases (for the most part). But I agree that the Oracle hire will be more difficult and costly to locate!

        I think you are right in that the OP might need someone to develop a solution. I may have misread the need --- I read it as a need to clean up the list they have but as you point out he may be looking for both the need to clean up the current list as well as a management solution for the future.
        {{ DiscussionBoard.errors[10134448].message }}
        • Profile picture of the author iAmNameLess
          Originally Posted by jamesfreddyc View Post

          I work with both Oracle and SQL Server... the SQL to filter as the OP wants is going be the same SQL statements across databases (for the most part). But I agree that the Oracle hire will be more difficult and costly to locate!

          I think you are right in that the OP might need someone to develop a solution. I may have misread the need --- I read it as a need to clean up the list they have but as you point out he may be looking for both the need to clean up the current list as well as a management solution for the future.
          You can do this using VTiger on any server, as long as you're not limited to queries. Might be better to run an install on your own system.

          After thinking about it, I think the obvious answer is PowerPivot with Excel, can handle 2 billion rows.... probably the cheapest solution as well as the lowest amount of hassle.
          Signature
          {{ DiscussionBoard.errors[10134587].message }}
          • Profile picture of the author savidge4
            If we open up the isolated problem needing a solution just a bit... First he needs to sort millions of entries. Then he has to work around injecting 50,000 entries at a time. and I am sure that "then" becomes another "then" and another "then".

            So you have raw separated data you need to bring together, sort, and batch to bring into a CRM that right off the bat is not working within the needs you are placing on it. Because I would ask this in real life.. Where else is this CRM solution failing?

            You can patch it, and patch It, and patch it till your blue in the face OR you can realize that you need a custom solution. If you think long term.. how much money are you going to spend ( have you spent already ) with patches before you spend the money to develop a solution that works for you?

            regardless of the platform there is an answer out there for you. It really is finding a programmer that can take your needs and develop a system that does what you need it to do. And hopefully on a platform that can be tweaked and manipulated down the road by someone other than the original programmer. ( I personally find that Oracle programmers note the crap out of everything and the programming style from programmer to programmer is very consistent - unlike any other platform I have worked with )
            Signature
            Success is an ACT not an idea
            {{ DiscussionBoard.errors[10134702].message }}
  • Profile picture of the author jamesfreddyc
    Oracle is overkill. SQL Server is a better option for the OP in this case. Far less costly and way more powerful than he would need.

    Heck, SQL Server Express would likely be workable and it's free. 2008R2 will grow to 10gb and for a basic set of tables that the OP is likely working with could be in the 10's of millions of rows.

    Install, load your data, send me a PM. I can send you the SQL statement to remove duplicates and merge rows. I can't do it for you, but I can provide some basic assistance. Beyond that you should look into hiring a tech person to help manage your data.

    I don't know if there's a cloud based solution to do it for you.
    {{ DiscussionBoard.errors[10134317].message }}
    • Profile picture of the author savidge4
      Originally Posted by jamesfreddyc View Post

      Oracle is overkill.

      With the money I have spent over the years in keeping up with training and the cost of the product... I aint using anything else.. over kill maybe... but I go from Excel to Oracle.. there is no middle ground LOL
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[10134765].message }}
  • Profile picture of the author DIABL0
    Originally Posted by Nysurance View Post

    We have numerous databases of millions of prospects for numerous vertices. Looking for a good software or cloud base solution that is cost effective in order to manage, merge, and dedupe all contacts. Most solutions we have found so far either can't handle that many records or are beyond our budget . Would appreciate any insight anyone could provide!
    What "exactly" are you looking to do?

    Does it actually need to be in a database VS flat file?

    I deal with 100's of millions of records and use flat files, as it is faster and simpler...been doing it that way for 15 years and use all custom built software. There's also some off the shelf stuff that may work for you, just not 100% sure what you are wanting to do.
    Signature
    How to Build LARGE EMAIL LISTS on a Budget and MONETIZE Like a PRO
    20+ Years Exp . . . . . . . . . . . . Email - CPA - PPL
    {{ DiscussionBoard.errors[10134541].message }}
  • Profile picture of the author Nysurance
    I really appreciate everyones valuable feedback . This has given me a lot of great info to use to create a solution. Thank you!
    {{ DiscussionBoard.errors[10136049].message }}

Trending Topics