Linux Three Musketeers (grep, awk, sed)

grep

• grep is a powerful command-line tool used to search for specified strings or regular expressions in files or text.
• Basic Syntax

grep [options] "search pattern" file

Search pattern: Can be a plain string or a regular expression.
File: Specifies the file(s) to search, which can be one or more files or input from a pipeline.

Common Options:

Examples:

• Display all lines containing "hello" in file.txt:

grep "hello" file.txt

• Search for "hello" in all files under /path/to/directory:

grep -r "hello" /path/to/directory

• Match "error" or "ERROR" regardless of case:

grep -i "error" log.txt

• Show matching lines along with line numbers:

grep -n "pattern" file.txt

• List all files containing "keyword" in the current directory:

grep -l "keyword" *.txt

• Show lines containing "important" along with 2 lines before and after:

grep -C 2 "important" document.txt

awk

• awk is a powerful text processing tool used for formatting, filtering, and analyzing data in files or text streams. It supports pattern-based matching and operations, making it useful for log analysis, data extraction, and simple scripting.
• Basic Syntax

awk 'pattern { action }' file

Pattern: The condition to match input data (optional).
Action: The operation to perform on matched data (optional).
If pattern is omitted, it matches all lines.
If action is omitted, it prints the matched lines by default.

Common Options:

Built-in Variables:

Examples:

• Print each line of a file:

awk '{ print $0 }' file.txt

• Print the first and third fields of each line:

awk '{ print $1, $3 }' file.txt

• Print the second field using a comma as the delimiter:

awk -F "," '{ print $2 }' data.csv

• Print lines where the first field is greater than 50:

awk '$1 > 50 { print $0 }' file.txt

• Print lines containing "error":

awk '/error/ { print $0 }' log.txt

• Print each line along with the number of fields it has:

awk '{ print $0, "Fields:", NF }' file.txt

• Print the total number of lines after processing:

awk 'END { print "Total lines:", NR }' file.txt

• Print "Processing started" before reading, the first field for each line, and "Processing finished" after:

awk 'BEGIN { print "Processing started" } { print $1 } END { print "Processing finished" }' file.txt

• Print lines where the first field is greater than 100 using a variable:

awk -v threshold=100 '$1 > threshold { print $0 }' file.txt

• Calculate the sum of the first field in the file:

awk '{ sum += $1 } END { print "Total:", sum }' file.txt

sed

• sed (Stream Editor) is a powerful tool for performing batch text processing, including search and replace, deletion, and insertion. It works with files or standard input text streams.
• Basic Syntax

sed [options] 'script command' file

Script command: Specifies the rule for text processing, such as replacement or deletion.
File: The file to process, or text can be passed through a pipeline.

Common Options:

Common Commands:

Examples:

• Replace the first occurrence of old with new in file.txt:

sed 's/old/new/' file.txt

• Delete the second line of file.txt:

sed '2d' file.txt

• Append "This is a new line" after the third line:

sed '3a\This is a new line' file.txt

• Insert "This is an inserted line" before the third line:

sed '3i\This is an inserted line' file.txt

• Replace the content of the second line with "This is the new content":

sed '2c\This is the new content' file.txt

• Print only lines containing "pattern":

sed -n '/pattern/p' file.txt

• Delete the first line and replace "foo" with "bar":

sed '1d; s/foo/bar/' file.txt

• Modify file.txt directly, replacing all occurrences of old with new:

sed -i 's/old/new/g' file.txt

• Delete lines 2 through 4:

sed '2,4d' file.txt

• Apply commands from script.sed to file.txt:

sed -f script.sed file.txt

These three tools—grep, awk, and sed—are essential for powerful text processing in Linux.