Mastering grep, sed, awk, curl, jq(For System Administrators)

1. Mastering grep

What grep does

Search inside files, filter logs, extract patterns.

Most important commands

grep "pattern" file
grep -i "pattern" file      # ignore case
grep -r "pattern" /path     # recursive
grep -v "pattern" file      # exclude
grep -E "regex" file        # extended regex
grep -A3 -B2 "error" file   # show context lines

Real-life examples

# 1. Find failed SSH logins
grep "Failed password" /var/log/auth.log

# 2. Extract IPs
grep -Eo "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" log.txt

# 3. Find all ERROR except INFO
grep "ERROR" app.log | grep -v "INFO"

2. Mastering sed

What sed does

Edit text in pipelines, replace config values, remove lines.

Most important commands

# Replace first match
sed 's/old/new/' file

# Replace globally
sed 's/old/new/g' file

# Edit file in-place
sed -i 's/old/new/' file

# Delete lines matching pattern
sed '/DEBUG/d' file

# Print only specific lines
sed -n '10,20p' file

Real‑life examples

# Change server port
sed -i 's/8080/9090/' server.conf

# Remove blank lines
sed -i '/^$/d' file

3. Mastering awk

What awk does

Parse text, process columns, create reports, calculate values, filter rows.

Awk basics

$0   = full line
$1   = first column
$2   = second column
NF   = number of fields
NR   = row number

Most important commands

awk '{print $1,$3}' file

# Filter rows
awk '$3 > 80 {print $0}' cpu.txt

# Add computed column
awk '{print $1, $2, $3*2}' file

# Sum a column
awk '{sum += $3} END {print sum}'

Real‑world examples

# Top repeated IPs
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

# Processes with high CPU
ps aux | awk '$3 > 50 {print $1,$2,$3}'

4. Mastering curl

What curl does

Make HTTP requests, test APIs, upload/download files.

Most important commands

curl https://api.github.com

curl -O http://example.com/file.txt

curl -X POST -d "name=admin" http://localhost/user

curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"name":"Ved"}' \
     https://api.example.com/users

curl -H "Authorization: Bearer TOKEN" https://api

5. Mastering jq

What jq does

Pretty print JSON, filter JSON, extract fields, work with API and Kubernetes output.

Most important commands

echo '{"a":1}' | jq

jq '.name'

jq '.data.items[].metadata.name'

jq '.users[] | select(.active==true)'

jq '.items | length'

Real‑world examples

kubectl get pods -o json | jq '.items[].metadata.name'

curl -s https://api.github.com/users/octocat | jq '.public_repos'

6. Combining all 5 tools

# Find top failed SSH IPs
grep "Failed password" /var/log/auth.log |
    awk '{print $11}' |
    sort | uniq -c | sort -nr | head

# Extract URLs from JSON API
curl -s https://api.github.com/repos/linux |
    jq '.owner.url'

# High CPU pods
kubectl top pods -A |
    grep -v kube-system |
    awk '$3 > 80 {print $0}'

7. Projects to Master These Tools

Project 1 — Log Analyzer

Identify top errors, IPs, slow requests.

Project 2 — API Automation

Call APIs, parse JSON, store results.

Project 3 — System Monitoring Script

CPU, RAM, network, disk using awk+grep.

Project 4 — Kubernetes Inspector

Extract node/pod metrics via jq.

Project 5 — Configuration Manager

Edit YAML, INI, CONF with sed+grep.