Process Text Stream with Filters

Text filters are used to modify the output of a file. They are helpful when searching for a specific keyword  and viewing a different output format from a file.

cat command is used to display the contents of a file. There are helpful options in viewing the file contents. The most commonly used are:
–   n: number each line of output
–   b: number only non-blank output line
–   A: show carriage return

Another function of cat is it can be used as a rudimentary text editor. How? By using a redirect. A redirect is used with a greater than sign >. Here’s a sample:

cat > sample-file
I'm typing the content of the sample-file.

Use CTRL+D to to save the content and end the interactive input.

You can also use the tac command to read text from the last to the first line. Obviously it’s the opposite of cat.

When analyzing log files, head or tail commands are helpful. By default, both output 10 lines of  text.

head file: outputs the first 10 lines
head [-20] file: outputs the first 20 lines
tail [-20] file: outputs the last 20 lines
head [+25] file: list text starting at line 25
tail -f file: continuously read a file, for real-time monitoring.

The wc utility for word count. It counts the number of bytes, words, and lines in files.
Options for wc output are:
–   l: count number of lines
–   w: count number of words
–   c or m: count number of bytes or characters

Another utility is nl. It functions like the cat -n.

nl -ba file: number lines including blanks
nl -bt file: number lines with text

The expand utility is used to replace TABs with spaces while the unexpand command is used for reverse operation.

To view binary files, use the hexdump utility. od utility may also be used.

To split a file into a smaller file, use the split utility.

split -1 5 file: create files xaa to xae w/ 5 lines each
split -1 5 file name-: create files xname-aa to name-ae

To not display the consecutive identical lines, use the uniq utility.

To extract a range of characters or fields from each line of text, use the cut utility.

cut - c range1, range 2 file: to manipulate characters
cut -d delimeter -f fields --output-delimeter=" " file

paste utility concatenates two files next to each other.

paste file1 file2

join utility outputs match fields between two files.

join -1 field_num -2 field_num file1 file2

sort utility arrange text in alphabetical order. -n option is used for numerical sort.

fmt utility to format output lines. fmt options are:
–   w: number of characters per line
–   s: split long lines but do not refill
–   u: place one space between each word and two spaces at the end of the sentence.

pr utility to paginate a file.

tr utility to translate one set of character ot another.

tr 'character 'character' < file

sed utility to search and replace patterns in text

sed '/ search_pattern/ new_pattern' file: substitute a pattern
sed 's/search_pattern/ new_pattern/g' file: substitute a pattern
 s - substitute
 g - globally; force substitution
sed '/key/new_pattern/g' file: if string exists, substitute a pattern

options for sed
–   e: execute following command
–   f: read commands from file
–   n: do not print out unedited lines
commands of sed
d – delete an entire line
r – read a file and append to output
s – substitute
w – write output to file