Shell Scripting Advanced: Automation, Logging, Backups & Live Projects

This comprehensive guide covers advanced DevOps shell scripting techniques required to build robust, failure-resistant automation pipelines. It explores battle-tested error-handling mechanisms, Makefile wrappers, and three hands-on real-world projects: AWS Cloud infrastructure auditing, automated backup and restore systems using Cron, and system log parsing and analysis.

1. Why Do We Need Advanced Shell Automation?

In modern cloud infrastructure, writing basic sequential terminal commands is insufficient for production workloads. Unhandled errors, unbounded log file growth, and unmonitored cloud resources lead to silent failures, unexpected cloud bills, and system downtime.

Structure & Concept Flow

Naive Scripting (Fragile)

+-- Fails silently on error

+-- Unbounded disk usage (no log management)

+-- Manual, error-prone execution

Production Scripting (Resilient)

+-- Explicit error traps & pipefail guards

+-- Automated execution & log rotation

+-- Self-documenting task drivers (Makefiles)

To transition from ad-hoc scripting to enterprise-grade automation, DevOps engineers rely on strict error flags, structured task runners, automated scheduling, and targeted text-parsing utilities.

2. Foundation: Robust Error Handling in Shell Scripts

By default, Bash continues executing subsequent commands even if a previous command returns a non-zero exit code (failure). In production environments, this can lead to data loss or corrupt states (e.g., attempting to compress a backup file when the database dump failed).

Essential Execution Flags

Production shell scripts must start with explicit execution flags:

BASH

#!/usr/bin/env bash
set -e          # Exit immediately if a command exits with a non-zero status
set -u          # Treat unset variables as an error and exit immediately
set -o pipefail # Pipeline exit status is that of the last command to fail

Note: set -e alone does not catch failures inside piped commands (e.g., command_fails | tee output.log). Combining it with set -o pipefail ensures that failures anywhere in a pipeline break script execution safely.

Trap Mechanics for Clean Teardown

The trap command intercepts signals (like SIGINT, EXIT, or error states) to execute cleanup routines before exiting.

BASH

#!/usr/bin/env bash
set -euo pipefail

TEMP_DIR=$(mktemp -d)

# Cleanup function to run on script exit regardless of success/failure
cleanup() {
    echo "[INFO] Cleaning up temporary directory: ${TEMP_DIR}"
    rm -rf "${TEMP_DIR}"
}

# Register the trap for EXIT signal
trap cleanup EXIT

# Script logic
echo "Processing data in ${TEMP_DIR}..."
# If any command fails past this point, cleanup() is guaranteed to run

3. Build & Task Automation with Makefiles

While shell scripts hold the core logic, invocation commands can become lengthy and difficult to remember. Wrapping shell scripts inside a Makefile provides a standard interface (make target) for developers and CI/CD pipelines.

`Makefile` Integration

Structure & Concept Flow

.PHONY: help build-aws backup parse-logs clean

# Default target

help:

@echo "Available commands:"

@echo " make build-aws - Execute AWS Cloud Resource Audit"

@echo " make backup - Trigger system backup job"

@echo " make parse-logs - Execute log analysis parser"

build-aws:

@echo "--> Running AWS Resource Audit..."

@bash ./scripts/aws_resource_tracker.sh

backup:

@echo "--> Running Backup Automation..."

@bash ./scripts/backup_restore.sh backup /var/www/html /backups

parse-logs:

@echo "--> Analyzing System Logs..."

@bash ./scripts/log_parser.sh /var/log/nginx/access.log

clean:

@echo "--> Cleaning up local temporary build files..."

@rm -rf ./tmp/*

4. Project 1: AWS Cloud Automation via Shell Scripting & AWS CLI

Problem Statement

Unused EC2 instances, untagged S3 buckets, stale IAM users, and abandoned Lambda functions accumulate in enterprise AWS accounts, causing billing spikes and compliance risks.

Architectural Workflow

Structure & Concept Flow

DevOps Admin / Cron / Makefile

aws_resource_tracker.sh

+--> AWS CLI (ec2 describe-instances) --> Formatted JSON / Text Report

+--> AWS CLI (s3api list-buckets) --> Formatted JSON / Text Report

+--> AWS CLI (iam list-users) --> Formatted JSON / Text Report

+--> AWS CLI (lambda list-functions) --> Formatted JSON / Text Report

Saved to Audit Log (/var/log/aws_audit.log)

Complete Production Script: `aws_resource_tracker.sh`

BASH

#!/usr/bin/env bash
###############################################################################
# Author: DevOps Team
# Description: Audits AWS Cloud Resources (EC2, S3, IAM, Lambda) and logs output.
###############################################################################

set -euo pipefail

# Configuration
LOG_FILE="/var/log/aws_resource_audit.log"
DATE_STAMP=$(date "+%Y-%m-%d %H:%M:%S")

# Redirect stdout and stderr to both log file and terminal
exec > >(tee -a "${LOG_FILE}") 2>&1

echo "=================================================="
echo " AWS Resource Audit Started At: ${DATE_STAMP} "
echo "=================================================="

# Check AWS CLI Installation
if ! command -v aws &> /dev/null; then
    echo "[ERROR] AWS CLI is not installed. Exiting."
    exit 1
fi

echo -e "\n1. Listing EC2 Instance IDs and States:"
aws ec2 describe-instances \
    --query "Reservations[*].Instances[*].[InstanceId, State.Name, InstanceType]" \
    --output table

echo -e "\n2. Listing S3 Buckets:"
aws s3api list-buckets --query "Buckets[*].Name" --output table

echo -e "\n3. Listing IAM Users:"
aws iam list-users --query "Users[*].[UserName, UserId, CreateDate]" --output table

echo -e "\n4. Listing Lambda Functions:"
aws lambda list-functions --query "Functions[*].[FunctionName, Runtime]" --output table

echo -e "\n=================================================="
echo " AWS Resource Audit Completed Successfully "
echo "=================================================="

5. Project 2: Automated Backup & Restore System with Cron Jobs

Problem Statement

Manual database and file backups are unreliable. A failure to take scheduled backups or clean up old archives can result in total data loss or disk exhaustion.

Real-Life Analogy

The Bank Safe Deposit Box: Taking a daily snapshot of files is like placing copies of critical documents into a numbered bank vault. To keep the vault from bursting, an automated policy destroys documents older than 30 days while leaving current copies readily available for restoration.

Execution Flow

Structure & Concept Flow

Cron Scheduler (Triggers Daily at 02:00 AM)

backup_restore.sh

+-- Create Timestamped Archive (.tar.gz)

+-- Generate MD5 / SHA256 Checksum Verification

+-- Store in Backup Destination

+-- Prune Backups Older Than Retention Threshold (e.g., 7 days)

Complete Production Script: `backup_restore.sh`

BASH

#!/usr/bin/env bash
###############################################################################
# Description: Automated Backup & Restore Utility with Retention Management
###############################################################################

set -euo pipefail

ACTION="${1:-}" # Mode: "backup" or "restore"
SOURCE_DIR="${2:-}"
DESTINATION_DIR="${3:-}"
RETENTION_DAYS=7

usage() {
    echo "Usage:"
    echo "  Backup:  $0 backup <source_dir> <backup_dir>"
    echo "  Restore: $0 restore <tar_archive> <restore_target_dir>"
    exit 1
}

if [[ -z "${ACTION}" \vert \vert  -z "${SOURCE_DIR}" || -z "${DESTINATION_DIR}" ]]; then
    usage
fi

case "${ACTION}" in
    backup)
        if [[ ! -d "${SOURCE_DIR}" ]]; then
            echo "[ERROR] Source directory ${SOURCE_DIR} does not exist."
            exit 1
        fi

        mkdir -p "${DESTINATION_DIR}"
        TIMESTAMP=$(date "+%Y%m%d_%H%M%S")
        ARCHIVE_NAME="backup_${TIMESTAMP}.tar.gz"
        TARGET_FILE="${DESTINATION_DIR}/${ARCHIVE_NAME}"

        echo "[INFO] Starting compression of ${SOURCE_DIR}..."
        tar -czf "${TARGET_FILE}" -C "${SOURCE_DIR}" .
        echo "[SUCCESS] Backup created at ${TARGET_FILE}"

        # Generate Checksum
        sha256sum "${TARGET_FILE}" > "${TARGET_FILE}.sha256"
        echo "[INFO] SHA256 checksum generated."

        # Retention Cleanup
        echo "[INFO] Pruning backups older than ${RETENTION_DAYS} days..."
        find "${DESTINATION_DIR}" -type f -name "backup_*.tar.gz" -mtime +"${RETENTION_DAYS}" -exec rm -f {} \;
        find "${DESTINATION_DIR}" -type f -name "backup_*.tar.gz.sha256" -mtime +"${RETENTION_DAYS}" -exec rm -f {} \;
        echo "[INFO] Retention cleanup complete."
        ;;

    restore)
        ARCHIVE_FILE="${SOURCE_DIR}"
        TARGET_RESTORE="${DESTINATION_DIR}"

        if [[ ! -f "${ARCHIVE_FILE}" ]]; then
            echo "[ERROR] Archive file ${ARCHIVE_FILE} does not exist."
            exit 1
        fi

        mkdir -p "${TARGET_RESTORE}"
        echo "[INFO] Restoring ${ARCHIVE_FILE} to ${TARGET_RESTORE}..."
        tar -xzf "${ARCHIVE_FILE}" -C "${TARGET_RESTORE}"
        echo "[SUCCESS] Restoration completed successfully."
        ;;

    *)
        usage
        ;;
esac

Scheduling via Cron

To run the backup script automatically every night at 2:00 AM:

BASH

# Open crontab editor
crontab -e

# Add the following entry:
0 2 * * * /usr/local/bin/backup_restore.sh backup /var/www/html /backups/web >> /var/log/cron_backup.log 2>&1

Cron Syntax Breakdown

Structure & Concept Flow

* * * * * command_to_execute

- - - - -

| | | | |

| | | | +----- Day of week (0 - 6) (Sunday=0)

| | | +------- Month (1 - 12)

| | +--------- Day of month (1 - 31)

| +----------- Hour (0 - 23)

+------------- Minute (0 - 59)

6. Managing Disk Usage: Log Rotation Strategies (`logrotate`)

Even with automated scripts, continuously accumulating logs will eventually consume all available inode/disk space. Linux provides logrotate to compress, rename, and recycle application logs automatically.

Configuring `/etc/logrotate.d/devops-app`

CONF

/var/log/devops_app/*.log {
    daily
    rotate 7
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/bin/systemctl reload devops_service > /dev/null 2>&1 || true
    endscript
}

Parameter Rationale

Directives	Purpose
`daily`	Rotates log files every single day.
`rotate 7`	Retains exactly 7 rotated log files before purging the oldest.
`missingok`	Suppresses errors if the target log file is missing.
`notifempty`	Skips log rotation if the file is 0 bytes (prevents empty file spam).
`compress`	Compresses older rotated logs using `gzip` to save disk space.
`delaycompress`	Postpones compression of the most recent log file to the next rotation cycle.
`postrotate`	Executes custom commands/scripts after the rotation is complete.

7. Project 3: Log Parsing, Filtering, and Analysis

Problem Statement

Production Web Servers generate millions of log entries daily. Locating specific server errors (HTTP 5xx status codes) or identifying suspicious traffic (brute-force IP addresses) manually using generic text editors is unfeasible.

Processing Complexity Analysis

Parsing an unindexed log file of $N$ lines using stream filtering tools (grep, awk, sed) executes in linear time complexity:

\mathcal{O}(N)

By chaining optimized Unix utilities through pipelines, system performance remains efficient even across multi-gigabyte log archives.

Pipeline Analysis Architecture

Structure & Concept Flow

Access Log (/var/log/nginx/access.log)

grep / awk (Extract 4xx/5xx HTTP status codes)

sort (Group similar IP addresses / endpoints)

uniq -c (Count occurrences)

sort -nr (Sort in descending numerical order)

head -n 10 (Display Top 10 offending IPs / endpoints)

Complete Production Script: `log_parser.sh`

BASH

#!/usr/bin/env bash
###############################################################################
# Description: Parses access logs to produce metric summaries and alert reports.
###############################################################################

set -euo pipefail

LOG_FILE="${1:-}"

if [[ -z "${LOG_FILE}" \vert \vert  ! -f "${LOG_FILE}" ]]; then
    echo "[ERROR] Valid log file path required."
    echo "Usage: $0 /path/to/access.log"
    exit 1
fi

echo "=================================================="
echo "          LOG ANALYSIS REPORT                     "
echo " Target File: ${LOG_FILE}"
echo " Date: $(date)"
echo "=================================================="

# 1. Total Requests Processed
TOTAL_REQUESTS=$(wc -l < "${LOG_FILE}")
echo -e "\nTotal Requests Processed: ${TOTAL_REQUESTS}"

# 2. Count HTTP Status Code Summary
echo -e "\nHTTP Status Code Summary:"
awk '{print $9}' "${LOG_FILE}" | grep -E '^[0-9]{3}$' | sort | uniq -c | sort -nr

# 3. Top 5 Client IP Addresses
echo -e "\nTop 5 Most Active Client IP Addresses:"
awk '{print $1}' "${LOG_FILE}" | sort | uniq -c | sort -nr | head -n 5

# 4. Top 5 Client Request Paths Returning 404 (Not Found)
echo -e "\nTop 5 Missing Endpoints (HTTP 404):"
awk '$9 == 404 {print $7}' "${LOG_FILE}" | sort | uniq -c | sort -nr | head -n 5

# 5. Extract Server Errors (HTTP 500, 502, 503, 504)
echo -e "\nCritical Server Errors (HTTP 5xx Summary):"
awk '$9 ~ /^5/ {print $0}' "${LOG_FILE}" > server_errors.tmp || true

SERVER_ERROR_COUNT=$(wc -l < server_errors.tmp)
echo "Total 5xx Failures Found: ${SERVER_ERROR_COUNT}"

if [[ "${SERVER_ERROR_COUNT}" -gt 0 ]]; then
    echo -e "\nSample 5xx Failure Logs:"
    head -n 3 server_errors.tmp
fi

rm -f server_errors.tmp

echo -e "\n=================================================="
echo " Analysis Complete "
echo "=================================================="

8. Complete Automation & Monitoring Flow

The diagram below illustrates how all components interoperate across a production infrastructure:

Structure & Concept Flow

Cron Daemon (Schedules Periodic Tasks)

+--> Triggers AWS Auditor Script

| |

| v

| AWS CLI Query API -> Generate Audit Logs

+--> Triggers Backup Script

| |

| v

| Tar/Gzip Source Folders -> Produce Archive & SHA Checksum

+--> Triggers Log Analyzer

Process /var/log/ App Logs -> Extract Top IPs & 5xx Alert Summaries

Logrotate Engine (Runs Nightly to Compress, Truncate, and Recycle Old Logs)

9. Real-World Production Pitfalls & Debugging Workflows

1. The Environment Variable Trap in Cron

Issue: A shell script runs successfully when triggered manually in the terminal, but fails silently when executed by Cron.
Root Cause: Cron runs inside a minimal subshell environment without standard user $PATH definitions (e.g., /usr/local/bin or AWS CLI binary paths might be missing).
Fix: Declare explicit PATH variables inside the script or provide absolute paths for executables:
BASH
```
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
```

2. Unhandled Pipe Failures

Issue: Using set -e fails to stop execution if a failed command is part of a pipeline.
Fix: Always include set -o pipefail at the start of scripts:
BASH
```
set -euo pipefail
```

3. Missing Execution Permissions

Issue: Attempting to execute scripts returns Permission denied.
Fix: Set executable permissions via chmod:
BASH
```
chmod +x scripts/*.sh
```

10. Comparative Analysis of DevOps Tooling & Strategies

Tool / Strategy	Primary Use Case	Strengths	Weaknesses / Limitations
Shell Scripts (Bash)	Low-level OS automation, system tasks, sequence orchestration.	Native execution on Linux, minimal dependencies, fast execution.	Verbose syntax for complex data structures, variable typing limitations.
Makefiles	Task driving, multi-step job wrappers, declarative target execution.	Standard interface (`make target`), cleanly hides lengthy commands.	Tab whitespace syntax sensitivity, not suited for heavy procedural logic.
Cron Jobs	Time-based background job scheduling.	Lightweight, built into Linux systems, reliable daemon execution.	Minimal default shell environment, lack of native distributed workflow tracking.
`logrotate`	File system retention, compression, disk management.	Prevents full disk failures automatically, highly configurable per service.	Requires correctly defined system user permissions and reload commands.
AWS CLI + Shell	Cloud infrastructure inspection, resource auditing, cleanup workflows.	Highly flexible API interaction, easy JSON output parsing via `jq`.	Requires properly scoped IAM credentials and proper error handling for network API failures.

11. Summary and Key Takeaways

Enforce Strict Error Modes: Always begin production shell scripts with set -euo pipefail to ensure immediate exit upon errors, unset variable access, or hidden pipe failures.
Implement Cleanup Traps: Use trap cleanup EXIT to handle temporary directory deletion and lockfile release reliably regardless of script exit conditions.
Wrap Workflows with Makefiles: Simplify command execution in development and CI/CD pipelines by wrapping complex shell scripts behind clean make targets.
Automate Infrastructure Audits: Use the AWS CLI inside shell scripts to query EC2, S3, IAM, and Lambda resources regularly to maintain cloud security hygiene and prevent unexpected billing costs.
Standardize Backup Protocols: Combine timestamped .tar.gz archive creation, SHA256 checksum generation, and find -mtime retention policies into automated backup workflows.
Schedule Tasks with Cron Explicitly: Account for Cron's restricted default environment by declaring absolute path variables (PATH) and redirecting standard output and error streams (>> logfile.log 2>&1).
**Prevent Disk Exhaustion via Log Rotation **: Implement system /etc/logrotate.d/ directives using daily, rotate, compress, and delaycompress to manage log file sizes safely.
Leverage Stream Processors for Analytics: Parse high-volume log files using linear-time $\mathcal{O}(N)$ Unix pipelines (awk, grep, sort, uniq -c) to isolate status codes and traffic spikes quickly.

Advanced: Automation, Logging, Backups

Shell Scripting Advanced: Automation, Logging, Backups & Live Projects

1. Why Do We Need Advanced Shell Automation?

2. Foundation: Robust Error Handling in Shell Scripts

Essential Execution Flags

Trap Mechanics for Clean Teardown

3. Build & Task Automation with Makefiles

Makefile Integration

4. Project 1: AWS Cloud Automation via Shell Scripting & AWS CLI

Problem Statement

Architectural Workflow

Complete Production Script: aws_resource_tracker.sh

5. Project 2: Automated Backup & Restore System with Cron Jobs

Problem Statement

Real-Life Analogy

Execution Flow

Complete Production Script: backup_restore.sh

Scheduling via Cron

Cron Syntax Breakdown

6. Managing Disk Usage: Log Rotation Strategies (logrotate)

Configuring /etc/logrotate.d/devops-app

Parameter Rationale

7. Project 3: Log Parsing, Filtering, and Analysis

Problem Statement

Processing Complexity Analysis

Pipeline Analysis Architecture

Complete Production Script: log_parser.sh

8. Complete Automation & Monitoring Flow

9. Real-World Production Pitfalls & Debugging Workflows

1. The Environment Variable Trap in Cron

2. Unhandled Pipe Failures

3. Missing Execution Permissions

10. Comparative Analysis of DevOps Tooling & Strategies

11. Summary and Key Takeaways

`Makefile` Integration

Complete Production Script: `aws_resource_tracker.sh`

Complete Production Script: `backup_restore.sh`

6. Managing Disk Usage: Log Rotation Strategies (`logrotate`)

Configuring `/etc/logrotate.d/devops-app`

Complete Production Script: `log_parser.sh`