awk, sed, cut and tr: processing a text file
When you want to process a text file using the command line, awk
, sed
, cut
and tr
are the most used programs. Here you will learn the basic use of each one.
Table of Contents
awk
This powerful tool can do simple and complex text processing tasks.
- Print (show) all contents of a file.
awk '{print}' text.txt #or awk '{print $0}' text.txt
Pipes
- You can pipe the output of another command to awk.
df -h | awk '{print $0}'
awk '{print}' < text.txt
- To Export the output to a file.
df -h | awk '{print}' > output.txt
Columns
- Print first column of a tab-separator file (or command).
df -h | awk '{print $1}'
- Print last column
awk '{print $NF}' text.txt
NF
stands for “Number of Fields”.
- Print several columns.
df -h | awk '{print $1,$3}'
- Using a diferent separator.
awk -F ":" '{print $1}' /etc/passwd
- You can add several code blocks inside the single quotes (
awk '{code block}; {code block}'
)
Rows
- Print a specific line
awk 'NR==3 {print}' text.txt
Formating the output
- Sum column values row by row (same for other mathematical operations).
$ awk -F ':' '{print $3+$4}' /etc/passwd 0 2 4
- Separate column values by some character (like a tab).
$ awk -F ',' '{print NR"\t"$1"\t"$2}' < test.csv 1 fechalectura lectura 2 2021-06-04 177 3 2021-08-10 184 4 2021-10-07 190
- Print the total number of lines (records).
# END: this code block runs when all input has been processed awk -F ':' 'END {print NR}' /etc/passwd
- Using
printf
.# Print second column as a floating number with two decimals awk -F ',' '{printf "%.2f\n",$2}' < testfile
Conditions
- Search for a RegExp.
# look for 'sshd' awk -F ':' '/sshd/ {print $1,$7}' /etc/passwd
- Standard “if”.
# Don't print first value of column 2 awk -F ',' '{if (NR != 1) print $2}' test.csv
# Split the file based on the third column value awk -F ',' '{if ($3 >= 1000) print $0}' < test.csv > testmore1000.csv awk -F ',' '{if ($3 < 1000) print $0}' < test.csv > testless1000.csv
Some examples
# Add a new calculated column and new column names
# BEGIN: this will run before file processing
# current, diff and prev are variables
$ awk -F ',' 'BEGIN {print "date,m3,diff"}; {current = $2}; {diff = current - prev}; {prev = $2}; {if (NR != 1) print $1","$2","diff}' ../file.csv | column -s ',' -t
date m3 diff
2021-06-04 177 177
2021-08-10 184 7
2021-10-07 190 6
2021-12-06 197 7
# Remove two lines
$ awk -F ',' '{if (NR==1 || NR==3) {} else {print NR " "$0}}' test.csv | head -n5
2 2021-07-21 00:00,247
4 2021-07-21 02:00,136
5 2021-07-21 03:00,82
6 2021-07-21 04:00,84
7 2021-07-21 05:00,115
# Search multiple patterns (AND)
$ ps aux | awk '/pts/ && /bash/'
ricardo 1521 0.0 0.0 11528 5892 pts/1 Ss abr19 0:00 /bin/bash
ricardo 9482 0.0 0.0 11132 5392 pts/39 Ss+ 12:45 0:00 /bin/bash
ricardo 15174 0.0 0.0 12224 3756 pts/1 S+ 14:02 0:00 awk /pts/ && /bash/
# Display a random line (look at the use of double quotes)
$ awk "NR==$(($RANDOM % `wc -l < urls.txt`))+1 {print}" urls.txt
https://eldiario.es
[ricardo@myarch tests]$ awk "NR==$(($RANDOM % `wc -l < urls.txt`))+1 {print}" urls.txt
https://google.com
sed
Use the ‘Streaming EDitor’ to transform a text.
Replace/delete text
- Substitute “word1” for “word2”.
sed 's/word1/word2/' text.txt
This command will not change the file, only show the results. You can export the output to a file with
>
or use-i
to edit the original file (use with caution, it’s always safer to create a new file).sed 's/word1/word2/' text.txt > newtext.txt
sed -i 's/word1/word2/' text.txt
Also, it only changes the first occurence in each line. To change all ocurrences:
sed 's/word1/word2/g' text.txt
sed 's/[[:lower:]]/\U&/g' lowertoupper.txt sed 's/[[:upper:]]/\l&/g' uppertolower.txt # Detects accented letters
- Delete word1.
sed 's/word1//g' text.txt
- Delete first character of every line.
sed 's/^.//' text.txt
- Delete last character of every line.
sed 's/.$//' text.txt
Replace “o” for “O” only on lines that match a pattern.
sed '/root/s/o/O/g' /etc/passwd
Delete lines
Delete lines matching a pattern.
sed '/root/d' /etc/passwd
You can use RegExp when looking for a pattern. This command will delete any empty line.
sed '/^$/d' test.txt
- Depending of the file formatting, you can also use
'/\r/d'
or'/^\r$/d'
.
Print matched lines
sed -n '/pattern/p' file.txt
Prepend/Append lines
Insert text one line before every line.
sed 'i\new line' test.txt
Append text (insert one line after every line).
sed 'a\new line' test.txt
Specify a line number
If you add a line number before the subcommand letter (a
, i
, d
, etc.) that subcommand will only run in that line. To refer to the last line, type $
.
# Delete second line
sed 2d test.txt
# Delete from line 3 to line 6
sed 3,6d test.txt
# Insert a line at the beginning
sed '1i/new line' test.txt
Extended RegExp support
For some RegExp, you may need to use -E
parameter.
sed -E '/(\.com)$/d'
Replace between patterns
You can use RegExp to split a line into several groups (using parenthesis) and replace only one of them.
$ cat urls.txt
https://elpais.com
https://eldiario.es
https://radiohuesca.com
https://google.com
$ sed -E 's/^(https:\/\/)(.*)(\.com)$/\1test\3/' urls.txt
https://test.com
https://eldiario.es
https://test.com
https://test.com
- You can select pattern groups with
\
and its number:1
for first group,2
for the second, etc. In this case, we want to print the first group (https://
), addtest
and print the third group (.com
).
More examples
-
Merge lines 2 and 3. Replace ‘2’ with the line you want to merge. You can add spaces or any character between the lines.
sed '2N;s/\n//' testfile
-
Change only first occurence in a file (from line 0 to
/loading/
)$ cat testfile <a href="/assets/imgs/test-1.jpg"><img src="/assets/imgs/test-1.jpg" alt="test" title="test"></a> <a href="/assets/imgs/test-2.jpg"><img src="/assets/imgs/test-2.jpg" alt="test" title="test"></a> $ sed '0,/loading/{s///}' testfile <a href="/assets/imgs/test-1.jpg"><img src="/assets/imgs/test-1.jpg" alt="test" title="test"></a> <a href="/assets/imgs/test-2.jpg"><img src="/assets/imgs/test-2.jpg" alt="test" title="test"></a>
cut
cut
is a simpler version of awk
. You can use it to separate a text in columns and show a specific column or several columns.
- Print first column on a “:” delimiter file.
cut -d ":" -f1 /etc/passwd
- Print two columns.
cut -d ":" -f1,7 /etc/passwd
- By default,
cut
uses the delimiter as a separator in the output, but you can change it with--output-delimiter=DELIMITER
.cut -d ":" -f1,7 --output-delimiter=" " /etc/passwd
tr
tr
works similar to sed
: translates or deletes characters from standard input, writing to standard output.
tr 'character' 'substitution' < file
For example, you can change commas to tabs:
tr ',' '\t' < file.csv
- You can achieve something similar with
column -s ',' -t < file.csv
.
Or you can delete all spaces:
echo {a..z} | tr -d ' '
Featured content: