15 {{ upvoteCount | shortNum }}
15 {{ upvoteCount | shortNum }}

How to Open and Split Huge Files

by BigFrank Banned
13 replies
Greetings. I have CSV files that are 200MB in size. No matter what I use to try and open them I get an error message stating the the files are "too large to open." Can someone tell me how I can open them and cut them into 40MB sized files?

I am on a Mac.

As an alternative, where can I find a service provider that I can pay to help me get this accomplished. TYVM.

Cheers. - Frank
#files #huge #open #split
  • Profile picture of the author Farish
    Upload to Google Drive, then export the file to Google Spreadsheet format. Should make it easier to manage and open.

    Or try OpenOffice if you weren't using that already to open them.

    200 megabytes for csv(text data) is really large.
    {{ DiscussionBoard.errors[8993463].message }}
    • Profile picture of the author BigFrank
      Banned
      Originally Posted by Farish View Post

      Upload to Google Drive, then export the file to Google Spreadsheet format. Should make it easier to manage and open.

      Or try OpenOffice if you weren't using that already to open them.
      Sounds like a plan. TYVM.

      200 megabytes for csv(text data) is really large.
      Yeah. It's around 2.5 million full business listings per file.

      Cheers. - Frank
      {{ DiscussionBoard.errors[8993587].message }}
  • Profile picture of the author jacktackett
    Frank, I've had to do this with some large data files and found this software to work great at splitting large files. Check out

    GSplit - File Splitter - Split Any File - Split Text and Log Files

    Best,
    Jack
    Signature
    Let's get Tim the kidney he needs!HELP Tim
    Mega Monster WSO for KimW http://ow.ly/4JdHm


    {{ DiscussionBoard.errors[8993811].message }}
    • Profile picture of the author BigFrank
      Banned
      Originally Posted by jacktackett View Post

      Frank, I've had to do this with some large data files and found this software to work great at splitting large files. Check out

      GSplit - File Splitter - Split Any File - Split Text and Log Files

      Best,
      Jack
      Hi Jack,

      I appreciate the suggestion but I believed I already looked into that and found that it was Windows, only. If worse comes to worse I may have to get a cheap windoze box.

      Not much out there in the Mac universe.

      Cheers. - Frank
      {{ DiscussionBoard.errors[8993892].message }}
      • Profile picture of the author Ankan333
        Originally Posted by BigFrank View Post

        Hi Jack,

        I appreciate the suggestion but I believed I already looked into that and found that it was Windows, only. If worse comes to worse I may have to get a cheap windoze box.

        Not much out there in the Mac universe.

        Cheers. - Frank
        Use wine to run win apps on mac.
        Signature

        If you are looking for a pro windows programmer then email me to : superbtheftauto@gmail.com

        {{ DiscussionBoard.errors[8994058].message }}
        • Profile picture of the author BigFrank
          Banned
          Originally Posted by Ankan333 View Post

          Use wine to run win apps on mac.
          Honestly, I'd rather just pay someone to do this for me. This whole process is well outside my comfort zone. It can't be that hard to find someone that can do this, easy-peasey. It ain't exactly rocket science.

          Cheers. - Frank
          {{ DiscussionBoard.errors[8994069].message }}
      • Profile picture of the author Farish
        Originally Posted by BigFrank View Post

        Hi Jack,

        I appreciate the suggestion but I believed I already looked into that and found that it was Windows, only. If worse comes to worse I may have to get a cheap windoze box.

        Not much out there in the Mac universe.

        Cheers. - Frank
        You don't need a windows box. You can run bootcamp to install and use Windows on your Mac(unless yours is an old power pc one.) or I use a program call Parallels to run Windows 8 virtually on my iMac.
        {{ DiscussionBoard.errors[8994376].message }}
        • Profile picture of the author BigFrank
          Banned
          Originally Posted by Farish View Post

          You don't need a windows box. You can run bootcamp to install and use Windows on your Mac(unless yours is an old power pc one.) or I use a program call Parallels to run Windows 8 virtually on my iMac.
          Hi. Yes - well aware of that capability, but at the risk of appearing snobbish, I'd sooner buy a windoze box than emulate one on my beloved Mac. lol I simply will not go there.

          Thanks and cheers. - Frank
          {{ DiscussionBoard.errors[8994973].message }}
  • Profile picture of the author valvednd
    i can help you out -- pretty cheap too. If there is a specific field you want the files chopped up on too i can split them anyway you need them... pm me if you are interested. if you are after file size that is easy enough too... what program are you using to open them with your mac ? 40mb may be too big still so i can further split them on a field (zip code/city or something) that is logical so the data is a little more managable.
    {{ DiscussionBoard.errors[8994237].message }}
    • Profile picture of the author BigFrank
      Banned
      Originally Posted by valvednd View Post

      i can help you out -- pretty cheap too. If there is a specific field you want the files chopped up on too i can split them anyway you need them... pm me if you are interested. if you are after file size that is easy enough too... what program are you using to open them with your mac ? 40mb may be too big still so i can further split them on a field (zip code/city or something) that is logical so the data is a little more managable.
      Hi. Thanks and I'll keep that in mind.

      Cheers. - Frank

      P.S. I was able to open them with MacVim, but they opened as a giant text file losing their formatting for importing into a DB. Not helpful.
      {{ DiscussionBoard.errors[8994979].message }}
  • Profile picture of the author valvednd
    cool, just a tip -- you should never concern your self with formatting within a database, if anything you should lower or upper everything and strip out number formatting for better querying... if you are planning to re-output the data you would want to handle specific formatting like that in your export script... you'll wind up running into weird issues with case sensitive languages or problems querying numbers if you try to retain the formatting from text data...
    {{ DiscussionBoard.errors[9001116].message }}
  • Profile picture of the author buysellbrowse
    How many lines can you read in your editor? In UNIX/Linux, you can use wc -l to find out how many lines there are in a file, then use the head and tail commands to split it into two parts. Further split the two halves if still too big.

    The MacOS is BSD based, and thus I presume it should have those commands as part of the OS.
    Signature
    buysell-browse.com * Free Classifieds Advertising & Promotion *
    {{ DiscussionBoard.errors[9002121].message }}
  • Profile picture of the author profitsforall
    I wouldn't recommend splitting based on file size - this may result in splitting part way through a line.

    This may be too advanced but you are running on a mac which is basically unix under the hood.

    There is a console application that allows you to type unix commands - not sure where it is as I'm on my pc.


    So you could change to the directory containing the file.

    e.g
    cd /path/to/your/file

    then type

    wc -l <yourfilename>

    This will tell you the number of lines in your file. For example it may come back and say you have 10000 lines. So you know 10000 lines = 200mb so 40mb would be 10000/5 (as 40 goes into 200 5 times). so 2000 lines.

    So to cut your file in to 2000 line chunks you would type

    split --lines=2000 <yourfilename>

    this will throw out a number of files I cant remember if it's named something like
    xa xb xc xd

    Horribly complicated if you don't know unix I guess. But thought I'd put it out there as an option.
    {{ DiscussionBoard.errors[9004192].message }}

Trending Topics