Option 1
cat bigFile.csv | parallel --header : --pipe -N999 'cat >file_{#}.csv'
References
Option 2
Part-1
In order to split your CSV file but keep the first header line, you need to create
a Shell script first.
- Open your code editor.
- Type in the below script and change the number of lines (currently set to 1000)
and file name (currently set to mycsvfile.csv)
accordingly. - Save the file in the same directory as your CSV file as split.sh.
#!/bin/bash
FILENAME=mycsvfile.csv
HDR=$(head -1 $FILENAME)
split -l 1000 $FILENAME xyz
n=1
for f in xyz*
do
if [ $n -gt 1 ]; then
echo $HDR > Part${n}.csv
fi
cat $f >> Part${n}.csv
rm $f
((n++))
done
Part-2
Once you’ve done this, open up Terminal on your Mac and navigate to your split.sh script
accordingly and run the sh command
(see below).
Note, it’s important to have a backup of your CSV file before running the below command:
sh split.sh