Chapter: Automated Backup and Log Rotation Scripts for DevOps

Automating system backups is a fundamental task in DevOps engineering. However, simply writing a script to compress files is not enough; a production-ready script must handle execution errors gracefully, suppress unnecessary console noise, and implement a rotation strategy to prevent disk space saturation. This chapter walks through the evolution of a robust backup and log rotation script, addressing common real-world bugs, error-checking mechanisms, and array-based file rotation.

1. The Problem: Why Do We Need Automated Backups & Rotation?

In production environments, applications continuously generate logs, user uploads, and database dumps. If backups are run daily without a cleanup strategy, the storage volume will eventually fill up, leading to system crashes and service downtime.

The Core Challenges

• Silent Failures: A script might print "Backup generated successfully" even if the underlying compression command failed due to permission issues or missing directories.
• Console Clutter: Standard output (stdout) and standard error (stderr) from utility commands (like zip or tar) can clutter cron job logs.
• Disk Saturation: Accumulating infinite backup files without a retention policy will inevitably exhaust disk space.

2. Core Mental Models & Analogies

To understand how a robust backup script operates, we can use two real-world analogies:

The "Black Hole" Redirection (/dev/null)

Real-Life Analogy: Imagine a physical office shredder. When you receive junk mail or warnings that you do not need to act upon, you drop them directly into the shredder instead of filing them on your desk. In Linux, /dev/null acts as this digital shredder. Any data redirected to it vanishes instantly without consuming disk space or cluttering your screen.

The "Filing Cabinet" Rotation Strategy

Real-Life Analogy: Imagine a physical filing cabinet that can only hold exactly 5 folders. When a new folder (the 6th one) arrives, you do not buy a new cabinet. Instead, you look at the dates on the folders, find the oldest one, shred it, and place the newest folder at the front. This keeps your storage footprint constant.

3. Step 1: Secure Variable Quoting & User Feedback

When writing shell scripts, variables containing file paths or timestamps must be handled carefully. If a directory path contains spaces (e.g., /home/ubuntu/my backups/), Bash will split the path into multiple arguments, causing the script to fail.

The Naive Approach (Unquoted Variables)

# Dangerous: If $item contains spaces, this command breaks
zip -r $backup_dir/$item.zip $source_dir

The Better Approach (Double Quoting)

To prevent word splitting and globbing issues, always wrap variable references in double quotes:

# Secure: Double quotes preserve spaces as a single string
zip -r "$BACKUP_DIR/$ITEM.zip" "$SOURCE_DIR"

4. Step 2: Robust Error Handling Using Exit Status ($?)

A common mistake in basic scripts is printing a success message immediately after a command, assuming it succeeded.

# Naive Flow
zip -r backup.zip /source/directory
echo "Backup generated successfully!" # This prints even if the zip command failed!

The Exit Status Variable ($?)

In Linux, every command returns an exit status (an integer between 0 and 255) upon completion:

• 0 indicates Success.
• Any non-zero value (e.g., 1, 2, 127) indicates a Failure or warning.

By checking the value of the special shell variable $? immediately after executing a command, we can conditionally print success or handle errors.

Implementation of Exit Status Check

# Execute the compression command
zip -r "$BACKUP_DIR/backup_$TIMESTAMP.zip" "$SOURCE_DIR"

# Check the exit status of the zip command
if [ $? -eq 0 ]; then
    echo "Backup generated successfully for timestamp: $TIMESTAMP"
else
    echo "Error: Backup generation failed!"
    exit 1
fi

5. Step 3: Silencing Warnings via I/O Redirection

When running automated scripts via cron jobs, utility commands like zip print verbose progress logs to the console. We can redirect this output to clean up the console and log files.

Standard Streams in Linux

• 0 : Standard Input (stdin)
• 1 : Standard Output (stdout)
• 2 : Standard Error (stderr)

Redirecting to /dev/null

To suppress standard output while still allowing critical errors to print, redirect descriptor 1 to /dev/null:

zip -r "$BACKUP_DIR/backup_$TIMESTAMP.zip" "$SOURCE_DIR" > /dev/null

To completely silence both standard output and standard error, redirect standard error (2) to standard output (&1):

zip -r "$BACKUP_DIR/backup_$TIMESTAMP.zip" "$SOURCE_DIR" > /dev/null 2>&1

6. Step 4: The Need for Backup Rotation

If a backup script runs daily, the backup directory will grow indefinitely. To prevent this, we must implement a Rotation Policy (e.g., Keep only the 5 most recent backups and delete the rest).

7. Step 5: Sorting Files Chronologically with ls -t

To identify which files are the oldest and which are the newest, we use the ls command with specific flags.

The ls -t Command

The -t flag sorts files by their modification time, showing the newest files first.

# Sort files in the backup directory by modification time (newest first)
ls -t /home/ubuntu/backups

Tabular Synthesis of ls Options for Scripting

8. Step 6: Implementing the perform_rotation Function in Bash

To implement rotation, we can read the sorted list of backups into a Bash array. Once the files are in an array, we can keep the first $N$ files (the newest ones) and delete any files beyond that index.

The Wildcard Path Bug

When storing file paths in an array, placing wildcards inside double quotes can prevent expansion:

# BUG: The wildcard * is treated as a literal character, causing "No such file or directory"
backups=("/home/ubuntu/backups/*.zip") 

The Correct Array Assignment

To resolve this, keep the wildcard outside of the double quotes or let the shell expand the command output:

# Correct: Command substitution expands the sorted list into an array
backups=( $(ls -t /home/ubuntu/backups/backup_*.zip) )

Array Slicing for Rotation

If we want to keep the 5 most recent backups, we can slice the array starting from index 5 (the 6th file) to get the list of files to delete:

# Syntax: ${array_name[@]:start_index}
files_to_delete=( "${backups[@]:5}" )

9. Complete Flow Walkthrough

The following vertical workflow illustrates the execution path of the complete script:

10. Complete Production-Ready Script

Below is the complete, fully documented Bash script containing both the backup creation logic and the automated rotation mechanism.

#!/bin/bash

###############################################################################
# Script Name:  backup_and_rotate.sh
# Description:  Automated backup creation with error checking and log rotation.
# Author:       DevOps Engineering Team
###############################################################################

# --- CONFIGURATION ---
SOURCE_DIR="/home/ubuntu/app/logs"
BACKUP_DIR="/home/ubuntu/backups"
TIMESTAMP=$(date "+%Y-%m-%d_%H-%M-%S")
BACKUP_FILE="$BACKUP_DIR/backup_$TIMESTAMP.zip"
RETENTION_COUNT=5

# Ensure backup directory exists
mkdir -p "$BACKUP_DIR"

# --- FUNCTION: CREATE BACKUP ---
create_backup() {
    echo "Starting backup of $SOURCE_DIR..."

    # Execute zip command and redirect standard output to /dev/null to keep console clean
    zip -r "$BACKUP_FILE" "$SOURCE_DIR" > /dev/null

    # Check exit status of the zip command
    if [ $? -eq 0 ]; then
        echo "Backup generated successfully: $BACKUP_FILE"
    else
        echo "Error: Backup generation failed!" >&2
        exit 1
    fi
}

# --- FUNCTION: PERFORM ROTATION ---
perform_rotation() {
    echo "Performing backup rotation (Retention Policy: Keep latest $RETENTION_COUNT)..."

    # Retrieve sorted list of backups (newest first)
    # We disable globbing checks for this command substitution expansion
    local backups=($(ls -t "$BACKUP_DIR"/backup_*.zip 2>/dev/null))

    # Check if total backups exceed our retention limit
    if [ "${#backups[@]}" -gt "$RETENTION_COUNT" ]; then
        echo "Found ${#backups[@]} backups. Rotating out older files..."

        # Slice array to get files starting from index $RETENTION_COUNT (0-indexed)
        local files_to_remove=("${backups[@]:$RETENTION_COUNT}")

        for file in "${files_to_remove[@]}"; do
            echo "Removing old backup: $file"
            rm -f "$file"
        done
        echo "Rotation completed successfully."
    else
        echo "Total backups (${#backups[@]}) within limit. No rotation needed."
    fi
}

# --- MAIN EXECUTION ---
create_backup
perform_rotation

11. Common Pitfalls & Debugging Workflows

1. The Script Prints "Success" but No File is Created

• Cause: The target directory path might be misspelled, or the parent directory lacks write permissions.
• Fix: Always check the exit status ($?) of the compression command before printing success. Ensure mkdir -p "$BACKUP_DIR" is run at the start of the script.

2. Wildcard Expansion Fails inside Arrays

• Cause: Writing backups=("path/*.zip") treats the asterisk (*) as a literal character rather than expanding it to match files.
• Fix: Use command substitution backups=($(ls -t path/*.zip)) or declare the array without double quotes around the wildcard: backups=( path/*.zip ).

3. Cron Job Fills Mailbox with Warnings

• Cause: Standard output and standard error are not redirected, causing the cron daemon to send emails containing the output of the zip command.
• Fix: Redirect output to /dev/null using > /dev/null 2>&1 inside the script.

Summary and Key Takeaways

• Always Quote Variables: Wrap file paths and string variables in double quotes ("$VAR") to prevent word splitting and globbing bugs.
• Validate with $?: Never assume a command succeeded. Use the exit status variable $? to verify execution before proceeding to subsequent steps.
• Silence Verbose Commands: Use I/O redirection (> /dev/null 2>&1) to discard unnecessary output and keep automated logs clean.
• Use ls -t for Chronological Sorting: Sorting files by modification time allows scripts to easily distinguish between new and old files.
• Leverage Bash Arrays for Rotation: Storing file lists in arrays and using slicing (${array[@]:N}) provides a clean, robust way to isolate and delete expired backups.