Shell Scripting Project: Automated Backup & Rotation with Cron Jobs

This chapter covers the implementation of an automated backup and rotation system using Bash scripting and Cron scheduling. You will learn how to write a script that packages directory contents, enforces a strict retention policy to prevent disk space exhaustion, and automates execution at regular intervals.

1. The Problem: Why Do We Need Backup Rotation?

In production environments, system administrators and DevOps engineers routinely back up critical application data, databases, and configuration files. However, a naive backup strategy that continuously generates files without cleanup introduces a critical vulnerability: disk space exhaustion.

If a backup script runs daily and saves a $1\text{ GB}$ archive, it will consume $365\text{ GB}$ of storage in a single year. If left unchecked, the host machine's disk usage will reach $100\%$, leading to:

• Database write failures.
• Application crashes due to the inability to write temporary files or logs.
• System-wide instability and downtime.

To prevent this, we must implement a Backup Rotation Policy. This policy defines a retention limit ($R$), ensuring that we only keep the $R$ most recent backups and automatically purge older archives.

2. Core Mental Model: The Sliding Window of Backups

The backup rotation mechanism operates on a Sliding Window (First-In, First-Out) model.

Real-Life Analogy

Think of a physical document organizer with exactly five slots. When you receive a new document, you place it in the first slot and slide the older documents down. When a sixth document arrives, the oldest document at the end of the organizer is discarded. This ensures you always have the five most recent documents without cluttering your desk.

Mathematical Representation

Let $B$ be the set of all existing backups ordered by creation time:

where $b_1$ is the oldest backup and $b_m$ is the newest. If our retention limit is $R$, we must maintain only the subset of backups:

Any backup $b_i$ where $i \le m - R$ must be identified and deleted. This keeps our storage footprint bounded at $O(R \times S)$, where $S$ is the average size of a single backup, rather than growing infinitely at $O(N \times S)$ over $N$ runs.

3. Implementing the Rotation Logic in Bash

The rotation script uses a for loop to iterate over an array containing the paths of the expired backups and deletes them using the rm -rf command.

Naive vs. Production-Ready Decoupled Code

The Naive Approach (Hardcoded and Fragile)

# Naive deletion without safety checks or dynamic array sizing
rm -rf /home/ubuntu/backups/backup_old.tar.gz

Why this is bad: It does not dynamically calculate which backups are old, relies on manual file naming, and risks deleting critical files if paths are misconfigured.

The Production-Ready Script (backup.sh)

Below is the complete, robust script that handles arguments, performs the backup, identifies expired archives, and executes the rotation loop safely.

#!/bin/bash

# ==============================================================================
# Script Name:  backup.sh
# Description:  Automates directory backups with a strict retention policy.
# ==============================================================================

# Ensure correct usage
if [ $# -ne 2 ]; then
    echo "Usage: $0 <source_directory_path> <backup_destination_path>"
    exit 1
fi

SOURCE_DIR="$1"
BACKUP_DIR="$2"
RETENTION_LIMIT=5

# Verify source directory exists
if [ ! -d "$SOURCE_DIR" ]; then
    echo "Error: Source directory '$SOURCE_DIR' does not exist."
    exit 1
fi

# Create backup destination directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Generate timestamped backup filename
TIMESTAMP=$(date "+%Y-%m-%d_%H-%M-%S")
BACKUP_FILENAME="backup_$TIMESTAMP.tar.gz"
BACKUP_FILEPATH="$BACKUP_DIR/$BACKUP_FILENAME"

# Create the compressed tarball
echo "Creating backup for: $SOURCE_DIR"
if tar -czf "$BACKUP_FILEPATH" -C "$SOURCE_DIR" . ; then
    echo "Backup generated successfully: $BACKUP_FILEPATH"
else
    echo "Error: Failed to create backup archive."
    exit 1
fi

# ------------------------------------------------------------------------------
# ROTATION LOGIC
# ------------------------------------------------------------------------------
echo "Performing rotation for $RETENTION_LIMIT days..."

# 1. List backups sorted by modification time (oldest first)
# 2. Store them in an array
backups=($(ls -1tr "$BACKUP_DIR"/backup_*.tar.gz 2>/dev/null))
total_backups=${#backups[@]}

if [ "$total_backups" -gt "$RETENTION_LIMIT" ]; then
    num_to_remove=$((total_backups - RETENTION_LIMIT))
    echo "Found $total_backups backups. Retention limit is $RETENTION_LIMIT. Removing $num_to_remove old backup(s)..."
    
    # Slice the array to get only the oldest files that exceed the retention limit
    backups_to_remove=("${backups[@]:0:$num_to_remove}")
    
    # Iterate and delete
    for backup in "${backups_to_remove[@]}"; do
        echo "Removing expired backup: $backup"
        rm -f "$backup"
    done
    echo "Rotation completed successfully."
else
    echo "Total backups ($total_backups) is within the retention limit ($RETENTION_LIMIT). No cleanup required."
fi

4. Automating Execution with Cron Jobs

To make this backup system production-grade, we must automate its execution so that it runs without manual intervention. In Unix-like operating systems, this is achieved using the Cron daemon.

What is a Cron Job?

A Cron job is a time-based job scheduler. It reads configuration files called crontabs (cron tables) to execute specified commands or scripts at predefined intervals.

The Cron Expression Syntax

A cron expression consists of five fields followed by the command to execute:

Cron Expression Reference Table

Tip: For complex schedules, use helper utilities like crontab.guru to construct and validate your cron expressions.

5. Step-by-Step Crontab Configuration

Follow these steps to configure your system to run the backup script automatically every minute.

Step 1: Open the Crontab Editor

Run the following command in your terminal:

crontab -e

If running this for the first time, the system will prompt you to select an editor. Select vim or nano (e.g., choose vim by entering its corresponding number).

Step 2: Add the Cron Entry

Navigate to the bottom of the file and append the following line:

* * * * * /bin/bash /home/ubuntu/backup.sh /home/ubuntu/data /home/ubuntu/backups

Breakdown of the Cron Entry:

• * * * * *: Instructs Cron to run the command every single minute.
• /bin/bash: Explicitly defines the shell interpreter to avoid environment path issues.
• /home/ubuntu/backup.sh: The absolute path to your backup script.
• /home/ubuntu/data: The Source Path argument passed to the script.
• /home/ubuntu/backups: The Backup Destination argument passed to the script.

Step 3: Save and Exit

• In Vim: Press Esc, type :wq, and press Enter.
• In Nano: Press Ctrl+O, Enter, then Ctrl+X.

Upon saving, you should see the confirmation message:

crontab: installing new crontab

6. Complete Flow Walkthrough & Verification

Once the cron job is installed, the system executes the workflow automatically.

Execution Flow Diagram

Verifying the Rotation in Real Time

To verify that the script is running and successfully rotating files, monitor the backup directory using the list command sorted by time (ls -lt or ls -1tr):

ls -1tr /home/ubuntu/backups

Initial State (Minutes 41 to 45):

As the cron job runs every minute, the directory populates up to the limit of 5 files:

backup_2024-10-24_14-41-00.tar.gz
backup_2024-10-24_14-42-00.tar.gz
backup_2024-10-24_14-43-00.tar.gz
backup_2024-10-24_14-44-00.tar.gz
backup_2024-10-24_14-45-00.tar.gz

State Change at Minute 46 (Rotation Triggered):

When the clock hits 14:46:00, a new backup is generated. The script detects that the total count is now 6, which exceeds the retention limit of 5. It automatically purges the oldest file (14-41-00):

Removing expired backup: /home/ubuntu/backups/backup_2024-10-24_14-41-00.tar.gz

Listing the directory again shows that the oldest backup has been deleted, and the total count remains exactly 5:

backup_2024-10-24_14-42-00.tar.gz
backup_2024-10-24_14-43-00.tar.gz
backup_2024-10-24_14-44-00.tar.gz
backup_2024-10-24_14-45-00.tar.gz
backup_2024-10-24_14-46-00.tar.gz

7. Real-World Production Pitfalls & Debugging

When deploying shell scripts to production via Cron, you may encounter common operational issues.

1. The "Command Not Found" or Path Issue

• Symptom: The script runs fine when executed manually, but fails or does nothing when run by Cron.
• Cause: Cron runs with a highly restricted, minimal environment. It does not load your .bashrc or .profile, meaning the standard PATH variable containing /usr/local/bin or user-specific paths is missing.
• Solution: Always use absolute paths inside your scripts and cron entries (e.g., use /bin/tar instead of tar, and /bin/bash instead of bash).

2. Permissions Denied

• Symptom: The cron log shows permission errors, or backups are not created.
• Solution: Ensure the script has execution permissions:
  BASH

  Copy

  chmod +x /home/ubuntu/backup.sh
  

  Additionally, verify that the user running the cron job has read permissions on the source directory and write permissions on the destination directory.

3. Silent Failures (No Logging)

• Symptom: The script fails silently, leaving no trace of what went wrong.
• Solution: Redirect standard output (stdout) and standard error (stderr) from your cron job to a log file for troubleshooting:
  TEXT

  Copy

  * * * * * /bin/bash /home/ubuntu/backup.sh /home/ubuntu/data /home/ubuntu/backups >> /home/ubuntu/backup.log 2>&1
  

Summary and Key Takeaways

• Disk Space Management: Unmanaged backups pose a severe risk of disk exhaustion. Implementing a retention policy is a fundamental requirement for production systems.
• Array Slicing in Bash: You can isolate a specific subset of files for deletion using Bash array slicing syntax: "{backups[@]:0:num_to_remove}".
• Automated Scheduling: Cron is the standard Unix utility for scheduling recurring tasks. Use crontab -e to manage schedules and crontab.guru to validate expressions.
• Absolute Paths: Always use absolute paths in cron configurations and production scripts to prevent environment-related execution failures.
• Error Handling: Production scripts must validate inputs (e.g., checking if the source directory exists with [ -d "$DIR" ]) and handle command failures gracefully to prevent data loss or silent failures.