Predictive Hacks

How to Concatenate Multiple CSV files in Linux

Assume that your files are my_file_1.csv, my_file_2.csv, my_file_3.csv and so on, and you want to concatenate all of them into one file. Then you can do it with the following command:

cat my_file_*.csv > merged.csv

which we generate a file called merged.csv.

How to deal with headers

Now, if your csv files have headers, and you want to keep the headers of just one file then you can use the following command:

awk '(NR == 1) || (FNR > 1)' my_file_*.csv > merged.csv

FNR refers to the number of the processed record in a single file and NR refers to all files, so we keep the first line which is the header and we ignore the first lines of each file.

Finally, if you want to remove the headers from all files:

awk 'FNR > 1' my_file_*.csv > merged.csv

Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

Subscribe To Our Newsletter

Get updates and learn from the best

More To Explore

pip
Python

Python pip Tips for Data Scientists

Data Scientists use to work with Anaconda Environments and for installing packages they use to run the “conda” commands. However,