Nice thing to use when working with CSV-files is CSV Kit.
See the file statistics: csvstat -e UTF-16 test.csv
Merge all files in a folder: csvstack -e UTF-16 -t *.csv > all.csv
Save column 2 to output .csv: csvcut -c 2 -e UTF-8 all.csv > output.csv
Remove duplicates:
Sort the file first: sort -n output.csv > sorted.csv
Save only unique values: uniq -u sorted.csv > unique.csv
Importing CSV data with text content
- Replace \" with ".
- Replace line breaks with empty space: ^$ with "".
- Use pipes "|" as field separator.
- Build your own text field with concat strings.
