Some helpful SED and AWK scripts

I regularly use some simple SED and AWK scripts to simplify life, but I tend to forget the specifics of their syntax, and have to start googling over and over and over in order to find something useful along the lines I want to have. SO here's a collection of stuff that I can easily copy/paste to my Terminal. It may not always be the best way to go about stuff, it simply works, so that's that. Don't complain or contact me with better solutions as I won't really care :)

Split a line into multiple lines

The following AWK oneliner splits a line into multiple lines, for a specific separator (":" in this case)

$ echo "text:text:text" | awk -v RS=":" 'length()==0{next}{print $0}' text text text

$

Merge multiple lines into one line

For merging a bunch of lines into a single line, here separated with a comma, use the following SED oneliner:

$ sed -e :a -e '$!N; s/\n/,/; ta' file

You can also combine 2, 3, 4, ... lines (add an extra N; for each line that needs to be merged) with the following SED script:

$ echo "hello world hello universe" | sed -e 'N;s/\n/,/g' hello,hello hello,hello $

Swap two columns

The following AWK one liner swaps two columns in an input file, and converts the input separator (FS) to a different output separator (OFS)

$ echo "hello|world > hello|universe" | awk 'BEGIN {FS="|"} {OFS="\t"} {print $2,$1}' world hello universe hello $

Conditionals in multiple columns

Awk can be used to "grep" in columns of a TSV file ...

Here's an example of getting information from a TSV file with various operators

awk '($2=="1") && ($3>=53565901) && ($3<=53566551)' file.tsv

Combining above

For a tab separated input with 3 columns, select those rows where the 3rd column is not "0".

awk 'BEGIN {FS="\t"} {OFS="\t"} {if($3!="0")print $1,$2,$3;}'

Remove top line from a file

sed '1d' file

Count the number of columns in a file

awk 'BEGIN {FS="\t"} ; END{print NF}' file


comments powered by Disqus