How to Open and Split Huge Files

by Banned 13 replies
15
Greetings. I have CSV files that are 200MB in size. No matter what I use to try and open them I get an error message stating the the files are "too large to open." Can someone tell me how I can open them and cut them into 40MB sized files?

I am on a Mac.

As an alternative, where can I find a service provider that I can pay to help me get this accomplished. TYVM.

Cheers. - Frank
#programming #files #huge #open #split
  • Upload to Google Drive, then export the file to Google Spreadsheet format. Should make it easier to manage and open.

    Or try OpenOffice if you weren't using that already to open them.

    200 megabytes for csv(text data) is really large.
    • [1] reply
    • Banned
      Sounds like a plan. TYVM.

      Yeah. It's around 2.5 million full business listings per file.

      Cheers. - Frank
  • Frank, I've had to do this with some large data files and found this software to work great at splitting large files. Check out

    GSplit - File Splitter - Split Any File - Split Text and Log Files

    Best,
    Jack
    • [1] reply
    • Banned
      Hi Jack,

      I appreciate the suggestion but I believed I already looked into that and found that it was Windows, only. If worse comes to worse I may have to get a cheap windoze box.

      Not much out there in the Mac universe.

      Cheers. - Frank
      • [2] replies
  • i can help you out -- pretty cheap too. If there is a specific field you want the files chopped up on too i can split them anyway you need them... pm me if you are interested. if you are after file size that is easy enough too... what program are you using to open them with your mac ? 40mb may be too big still so i can further split them on a field (zip code/city or something) that is logical so the data is a little more managable.
    • [1] reply
    • Banned
      Hi. Thanks and I'll keep that in mind.

      Cheers. - Frank

      P.S. I was able to open them with MacVim, but they opened as a giant text file losing their formatting for importing into a DB. Not helpful.
  • cool, just a tip -- you should never concern your self with formatting within a database, if anything you should lower or upper everything and strip out number formatting for better querying... if you are planning to re-output the data you would want to handle specific formatting like that in your export script... you'll wind up running into weird issues with case sensitive languages or problems querying numbers if you try to retain the formatting from text data...
  • How many lines can you read in your editor? In UNIX/Linux, you can use wc -l to find out how many lines there are in a file, then use the head and tail commands to split it into two parts. Further split the two halves if still too big.

    The MacOS is BSD based, and thus I presume it should have those commands as part of the OS.
  • I wouldn't recommend splitting based on file size - this may result in splitting part way through a line.

    This may be too advanced but you are running on a mac which is basically unix under the hood.

    There is a console application that allows you to type unix commands - not sure where it is as I'm on my pc.


    So you could change to the directory containing the file.

    e.g
    cd /path/to/your/file

    then type

    wc -l <yourfilename>

    This will tell you the number of lines in your file. For example it may come back and say you have 10000 lines. So you know 10000 lines = 200mb so 40mb would be 10000/5 (as 40 goes into 200 5 times). so 2000 lines.

    So to cut your file in to 2000 line chunks you would type

    split --lines=2000 <yourfilename>

    this will throw out a number of files I cant remember if it's named something like
    xa xb xc xd

    Horribly complicated if you don't know unix I guess. But thought I'd put it out there as an option.

Next Topics on Trending Feed

  • 15

    Greetings. I have CSV files that are 200MB in size. No matter what I use to try and open them I get an error message stating the the files are "too large to open." Can someone tell me how I can open them and cut them into 40MB sized files? I am on a Mac.