Shell Scripting for DevOps: Environment Setup, AWS Automation, Log Processing, and Cron Jobs

Shell scripting is a foundational capability in DevOps engineering. It bridges the gap between manual CLI interactions and scalable, automated infrastructure orchestration. This guide covers shell scripting essentials, strict execution modes, text processing tools, and three production-ready DevOps projects: an AWS Resource Usage Reporter, an Automated Log Analysis & Archival Pipeline, and a Cron-based Task Scheduler.

1. Introduction to Shell Scripting and DevOps Automation

In a modern cloud and DevOps environment, manual execution of command-line operations is error-prone, non-repeatable, and unscalable. Shell scripts allow DevOps engineers to glue together disparate operating system utilities, cloud provider CLI tools, and system services into repeatable pipelines.

Structure & Concept Flow

Manual Operations (High Error Risk, Non-Repeatable)

CLI Commands Executed in Interactive Terminal

Shell Scripts (Automated, Idempotent, Version-Controlled)

Scheduled Jobs / CI/CD Pipeline Execution (Jenkins, GitHub Actions, Cron)

Why Shell Scripting Remains Essential

Zero External Dependencies: Bash is pre-installed across almost all Linux distributions (Ubuntu, RHEL, Alpine, Debian) and macOS.
Direct Kernel & OS Interface: Interfacing with processes, file systems, network sockets, and environment variables requires minimal abstraction.
Glue Language for Cloud Tools: Seamlessly chains output from tools like aws-cli, kubectl, docker, curl, and jq.

2. Architecture & Mental Model: Shebang, Shells, and Execution Flow

When a shell script is executed, the Linux kernel needs to know which interpreter to invoke to execute the commands contained within the file.

The Shebang (`#!`)

The first line of a shell script must begin with a shebang sequence: #!/bin/bash or #!/usr/bin/env bash.

Real-Life Analogy

Think of the kernel as a master kitchen head chef and the script as a customer's order ticket. The Shebang (#!/bin/bash) is the language identifier at the top of the ticket telling the kitchen whether the order must be prepared by the French chef (/bin/bash), the Italian chef (/usr/bin/python3), or the Japanese chef (/bin/zsh). Without it, the head chef might guess incorrectly and ruin the order.

Execution Process Flow

Structure & Concept Flow

User / Automation Agent

| invokes ./script.sh

Linux Kernel reads first line (Shebang)

+---> #!/bin/bash identified

Kernel spawns subshell process: /bin/bash ./script.sh

Subshell reads commands line-by-line

Child processes spawned for external commands (grep, aws, tar)

Script completes and returns Exit Code ($?) to parent process

3. Setup & Strict Script Mode Initialization

Default Bash behavior is notoriously forgiving: it ignores failed commands, uses uninitialized variables as empty strings, and swallows errors in piped commands. Production scripts must explicitly enforce Strict Mode.

Production Script Header Template

BASH

#!/usr/bin/env bash

# Enable strict error handling
set -e          # Exit immediately if a command exits with a non-zero status
set -u          # Treat unset variables as an error and exit
set -o pipefail # Return value of a pipeline is the status of the last command to exit with a non-zero status
# set -x        # Uncomment for debug mode (prints each command before execution)

# Environment Path Standardization
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

Note: Combining these flags as set -euo pipefail is widely considered the gold standard for writing resilient, fail-fast production shell scripts.

4. Text Processing Utilities Matrix

DevOps shell scripting heavily relies on text manipulation and stream parsing. The table below compares the standard core utilities used in shell pipelines:

Utility	Primary Purpose	Optimal Use Case	Time Complexity
`grep`	Line matching / Filtering	Filtering logs by error keywords or status codes	$O(N)$ scanning linear streams
`awk`	Column processing & structured reports	Extracting specific log fields, aggregating numbers	$O(N)$ row-by-row field processing
`sed`	Stream editing / Text substitution	In-place config file modifications, find-and-replace	$O(N)$ single-pass string editing
`jq`	JSON parsing and querying	Processing API responses from AWS CLI or REST endpoints	$O(N)$ tree parsing of JSON payloads
`cut`	Simple character/delimiter splitting	Splitting fixed-format delimited strings (`/etc/passwd`)	$O(N)$ byte/character cutting

5. Project 1: AWS Resource Usage Reporter

Business Problem

A DevOps team needs an automated audit script that queries AWS accounts daily and extracts active resources across EC2, S3, Lambda, and IAM to prevent idle cloud costs and track infrastructure sprawl.

Complete Production Code: `aws_resource_tracker.sh`

BASH

#!/usr/bin/env bash

###############################################################################
# Author: DevOps Team
# Date: 2026-03-30
# Version: v1.0.0
# Description: Reports AWS resource usage (S3, EC2, Lambda, IAM Users)
###############################################################################

set -euo pipefail

# Output log file location
REPORT_FILE="/var/log/aws_resource_report_$(date +%Y%m%d_%H%M%S).log"

# Redirect stdout and stderr to both console and report file
exec > >(tee -a "${REPORT_FILE}") 2>&1

echo "=================================================="
echo " AWS Resource Usage Report"
echo " Execution Timestamp: $(date)"
echo "=================================================="

# Function to check if AWS CLI is installed
check_aws_cli() {
    if ! command -v aws &> /dev/null; then
        echo "[ERROR] AWS CLI is not installed or not in PATH. Exiting." >&2
        exit 1
    fi
}

# 1. List S3 Buckets
list_s3_buckets() {
    echo ""
    echo "--------------------------------------------------"
    echo " 1. Listing AWS S3 Buckets"
    echo "--------------------------------------------------"
    aws s3 ls || echo "[WARNING] Failed to fetch S3 buckets."
}

# 2. List EC2 Instances (Instance ID, Type, State)
list_ec2_instances() {
    echo ""
    echo "--------------------------------------------------"
    echo " 2. Listing AWS EC2 Instances"
    echo "--------------------------------------------------"
    aws ec2 describe-instances \
        --query "Reservations[*].Instances[*].[InstanceId, InstanceType, State.Name, PublicIpAddress]" \
        --output table || echo "[WARNING] Failed to fetch EC2 instances."
}

# 3. List AWS Lambda Functions
list_lambda_functions() {
    echo ""
    echo "--------------------------------------------------"
    echo " 3. Listing AWS Lambda Functions"
    echo "--------------------------------------------------"
    aws lambda list-functions \
        --query "Functions[*].[FunctionName, Runtime, MemorySize, LastModified]" \
        --output table || echo "[WARNING] Failed to fetch Lambda functions."
}

# 4. List IAM Users
list_iam_users() {
    echo ""
    echo "--------------------------------------------------"
    echo " 4. Listing AWS IAM Users"
    echo "--------------------------------------------------"
    aws iam list-users \
        --query "Users[*].[UserName, UserId, CreateDate]" \
        --output table || echo "[WARNING] Failed to fetch IAM users."
}

# Main Execution Flow
main() {
    check_aws_cli
    list_s3_buckets
    list_ec2_instances
    list_lambda_functions
    list_iam_users
    echo ""
    echo "=================================================="
    echo " Report Generated Successfully: ${REPORT_FILE}"
    echo "=================================================="
}

main

6. Project 2: Automated Log Analysis & Archival Pipeline

Business Problem

Web servers generate gigabytes of log files daily. Unattended logs fill up disk mounts, leading to outages. The DevOps team requires a script to scan web access logs, summarize critical errors (5xx status codes and ERROR alerts), archive logs older than 7 days, and purge archives older than 30 days.

Complete Production Code: `log_analyzer.sh`

BASH

#!/usr/bin/env bash

set -euo pipefail

LOG_DIR="/var/log/nginx"
ARCHIVE_DIR="/var/log/nginx/archive"
RETENTION_DAYS=30
ARCHIVE_THRESHOLD_DAYS=7
SUMMARY_EMAIL="devops-alerts@company.com"

mkdir -p "${ARCHIVE_DIR}"

echo "[INFO] Starting Log Analysis and Maintenance Pipeline: $(date)"

# Step 1: Parse and Analyze Current Access Log for Errors
CURRENT_LOG="${LOG_DIR}/access.log"

if [[ -f "${CURRENT_LOG}" ]]; then
    echo "--------------------------------------------------"
    echo " Analyzing Critical Errors in ${CURRENT_LOG}"
    echo "--------------------------------------------------"
    
    # Count total 5xx Server Errors using grep and awk
    TOTAL_5XX=$(grep -E ' HTTP/1\.[01]" 5[0-9]{2} ' "${CURRENT_LOG}" | wc -l || true)
    
    # Count total 4xx Client Errors
    TOTAL_4XX=$(grep -E ' HTTP/1\.[01]" 4[0-9]{2} ' "${CURRENT_LOG}" | wc -l || true)
    
    # Extract Top 5 Requesting IP Addresses causing 5xx errors
    TOP_OFFENDERS=$(grep -E ' HTTP/1\.[01]" 5[0-9]{2} ' "${CURRENT_LOG}" | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 5 || true)

    echo "HTTP 5xx Server Error Count: ${TOTAL_5XX}"
    echo "HTTP 4xx Client Error Count: ${TOTAL_4XX}"
    echo "Top IPs causing 5xx errors:"
    echo "${TOP_OFFENDERS}"
    
    # Alert Trigger Logic
    if [[ "${TOTAL_5XX}" -gt 50 ]]; then
        echo "[CRITICAL] High error rate detected (${TOTAL_5XX} 5xx errors). Sending notification."
        # Email trigger placeholder:
        # echo "High 5xx errors: ${TOTAL_5XX}" \vert  mail -s "ALERT: Web Server High Error Rate" "${SUMMARY_EMAIL}"
    fi
else
    echo "[WARNING] Log file ${CURRENT_LOG} not found. Skipping analysis."
fi

# Step 2: Compress Logs Older than Threshold (7 Days)
echo "--------------------------------------------------"
echo " Archiving logs older than ${ARCHIVE_THRESHOLD_DAYS} days"
echo "--------------------------------------------------"
find "${LOG_DIR}" -maxdepth 1 -type f -name "*.log" -mtime +"${ARCHIVE_THRESHOLD_DAYS}" -exec gzip -v {} \;

# Move gzipped files to Archive Directory
find "${LOG_DIR}" -maxdepth 1 -type f -name "*.gz" -exec mv {} "${ARCHIVE_DIR}/" \;

# Step 3: Purge Archives Older than Retention Limit (30 Days)
echo "--------------------------------------------------"
echo " Purging log archives older than ${RETENTION_DAYS} days"
echo "--------------------------------------------------"
find "${ARCHIVE_DIR}" -type f -name "*.gz" -mtime +"${RETENTION_DAYS}" -print -delete

echo "[INFO] Log Pipeline Completed Successfully."

7. Project 3: Scheduling via Cron Jobs and Crontab

To transform manual shell scripts into automated background tasks, Linux relies on the Cron daemon.

Cron Syntax Architecture

Structure & Concept Flow

* * * * * /path/to/script.sh

| | | | |

| | | | +--- Day of Week (0 - 6) (Sunday=0 or 7)

| | | +------ Month (1 - 12)

| | +--------- Day of Month (1 - 31)

| +------------ Hour (0 - 23)

+--------------- Minute (0 - 59)

Common Cron Schedule Patterns

Schedule Syntax	Interpretation	Use Case
`0 * * * *`	Every hour on the hour	Dynamic DNS updates, shallow log checks
`0 0 * * *`	Daily at midnight ( $00:00$ )	Daily backup routines, log rotation
`0 6 * * 1-5`	At 06:00 AM, Monday through Friday	Pre-business infrastructure spin-up
`/15 * * *`	Every 15 minutes	Health checks, metrics scraping
`@reboot`	Once upon system boot	Daemon init, tunnel establishment

Cron Environment Traps & Solutions

Real-Life Analogy

Running a script manually in your interactive terminal is like driving your own car with all your personal items, keys, and GPS preferences loaded. Running a script via Cron is like handing the keys to a valet who has never sat in your car before: if you don't explicitly hand them the exact map (absolute paths) and environment settings (PATH), they won't know where anything is located and will stall immediately.

The 3 Golden Rules of Cron Automation:

Always Use Absolute Paths: Never use relative paths like ./script.sh or python script.py. Use /usr/bin/python3 /home/ubuntu/scripts/script.py.
Set the Environment explicitly: Define PATH inside the crontab or at the top of your shell script.
Capture Output Streams: Cron swallows stdout and stderr unless explicitly redirected to a file or log collector.

Setting Up Crontab for Projects

To open the user crontab editor:

BASH

crontab -e

Add the following production schedules:

BASH

# Set PATH explicitly inside Crontab
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Run AWS Resource Reporter every day at 01:00 AM and capture log
0 1 * * * /home/ubuntu/scripts/aws_resource_tracker.sh >> /var/log/cron_aws_tracker.log 2>&1

# Run Log Analyzer & Archival Pipeline every Sunday at midnight
0 0 * * 0 /home/ubuntu/scripts/log_analyzer.sh >> /var/log/cron_log_analyzer.log 2>&1

8. Failure Modes, Debugging & Signal Traps

Production-grade scripts must handle unexpected terminations (e.g., SIGINT, SIGTERM, keyboard interrupts, or command failures gracefully).

Signal Traps and Cleanup Routines

Use the trap command to catch signals and cleanup temporary resources before exiting.

BASH

#!/usr/bin/env bash
set -euo pipefail

# Create a temporary working directory
TMP_DIR=$(mktemp -d /tmp/devops_job_XXXXXX)

# Cleanup function definition
cleanup() {
    echo "[INFO] Cleaning up temporary files from ${TMP_DIR}..."
    rm -rf "${TMP_DIR}"
}

# Trap signals: EXIT, INT (Ctrl+C), TERM (kill)
trap cleanup EXIT INT TERM

# Script logic operating on temporary files
echo "Processing data in ${TMP_DIR}..."
echo "Sample telemetry data" > "${TMP_DIR}/data.txt"

# Even if an error happens or script exits, cleanup() triggers automatically.

Static Analysis with ShellCheck

Before deploying any script to production servers or CI/CD pipelines, validate code syntax using shellcheck:

BASH

# Install ShellCheck
sudo apt-get install -y shellcheck

# Run static analysis
shellcheck aws_resource_tracker.sh

9. Comparative Synthesis: Shell Tools & Scripting Patterns

Structure & Concept Flow

Raw Input Data Stream

+---------------+---------------+

| |

Structured JSON Unstructured Text Logs

| |

v v

Use `jq` Use `grep` / `awk` / `sed`

| |

+---------------+---------------+

Automated Shell Pipeline

Cron Scheduler / CI Execution

10. Summary and Key Takeaways

Enforce Strict Execution: Always begin production Bash scripts with set -euo pipefail to ensure immediate fail-fast termination on errors, uninitialized variables, or broken pipes.
Shebang Precision: Use #!/usr/bin/env bash for maximum portability across different Linux distributions and Unix-like operating systems.
AWS Automation: Combine aws-cli with --query projections or pipe output into jq to create light, zero-dependency cloud inventory reporters.
Log Maintenance Rules: Implement multi-stage log management pipelines—extract critical error signals immediately, compress historical logs with gzip, and purge archives beyond retention limits using find -mtime.
Cron Absolute Paths: Cron executes within a stripped-down environment. Always declare absolute system paths for binaries, configurations, and log outputs within crontab files.
Signal Handling: Implement trap hooks to ensure temporary working directories, lock files, and socket connections are cleaned up upon script exit or signal termination (SIGINT/SIGTERM).
Static Verification: Always run shellcheck against your scripts during local development and within CI/CD syntax verification steps to identify bug traps, quote mismatches, and execution edge cases.

Projects: AWS, Logs, Cron Jobs

Shell Scripting for DevOps: Environment Setup, AWS Automation, Log Processing, and Cron Jobs

1. Introduction to Shell Scripting and DevOps Automation

Why Shell Scripting Remains Essential

2. Architecture & Mental Model: Shebang, Shells, and Execution Flow

The Shebang (#!)

Real-Life Analogy

Execution Process Flow

3. Setup & Strict Script Mode Initialization

Production Script Header Template

4. Text Processing Utilities Matrix

5. Project 1: AWS Resource Usage Reporter

Business Problem

Complete Production Code: aws_resource_tracker.sh

6. Project 2: Automated Log Analysis & Archival Pipeline

Business Problem

Complete Production Code: log_analyzer.sh

7. Project 3: Scheduling via Cron Jobs and Crontab

Cron Syntax Architecture

Common Cron Schedule Patterns

Cron Environment Traps & Solutions

Real-Life Analogy

The 3 Golden Rules of Cron Automation:

Setting Up Crontab for Projects

8. Failure Modes, Debugging & Signal Traps

Signal Traps and Cleanup Routines

Static Analysis with ShellCheck

9. Comparative Synthesis: Shell Tools & Scripting Patterns

10. Summary and Key Takeaways

The Shebang (`#!`)

Complete Production Code: `aws_resource_tracker.sh`

Complete Production Code: `log_analyzer.sh`