Managing Services & Background Jobs (systemctl, nohup) & Automated Backup Rotation

Automating system administration tasks is a core responsibility in DevOps. A common real-world task is managing system backups. However, if backups are allowed to accumulate indefinitely, they will eventually consume all available disk space, leading to system failures.

This chapter covers the implementation of an automated Backup Rotation Script in Bash. It explains how to safely manage backup files using Bash arrays, conditional logic, and array slicing. Additionally, it covers how to run these scripts reliably in production environments using background job utilities (nohup) and system service managers (systemctl).

1. The Problem: Uncontrolled Backup Accumulation

In a production environment, automated cron jobs regularly generate backup archives (e.g., .tar.gz files). Over time, these files pile up:

Without a rotation policy, the storage disk will eventually reach $100\%$ capacity, causing database crashes and service outages.

The Goal

We must design a script that:

1. Keeps only the 5 most recent backups.
2. Identifies and deletes all older backups beyond the threshold of 5.
3. Suppresses unnecessary error messages during execution.
4. Verifies the target files before deletion to prevent accidental data loss.

2. Core Mental Model: Array Slicing

To manage backups programmatically, we store the list of backup files in a sorted Bash array. By sorting the files by modification time (newest to oldest), we can use array slicing to separate the files we want to keep from the files we want to delete.

Real-Life Analogy: The Stack of Plates

Imagine a stack of plates where you only have room to store 5 plates on your shelf.

If you receive new plates, you place them on top. Any plate pushed below the 5th position (Index 5 and beyond) must be discarded to save space.

Mathematical Representation of Array Slicing

Let $A$ be an array of backups of length $N$, indexed from $0$ to $N-1$:

If $N > 5$, we partition the array into two subsets:

• Keep Set: $\{b_i \mid 0 \le i < 5\}$
• Removal Set: $\{b_i \mid 5 \le i < N\}$

In Bash, this partition is achieved using the slice syntax: ${array_name[@]:start_index}.

3. Step-by-Step Bash Implementation

Step 3.1: Suppressing Standard Errors (2>/dev/null)

When querying directories or executing commands inside automation scripts, non-critical errors (such as searching an empty directory or encountering permission restrictions on specific files) can pollute the logs.

To prevent this, we redirect Standard Error (stderr) to /dev/null (the system black hole):

# Redirecting stderr to discard unwanted error messages
ls -t /home/backups 2>/dev/null

Step 3.2: Checking Array Length

Before performing any deletion, we must check if the total number of backups exceeds our threshold of 5. If we have 5 or fewer backups, no rotation is needed.

In Bash, the length of an array is accessed using the ${#array_name[@]} syntax:

# Check if the length of the backups array is greater than 5
if [ "${#backups[@]}" -gt 5 ]; then
    echo "Performing rotation for backups..."
fi

Step 3.3: Extracting Old Backups via Array Slicing

To isolate the backups that need to be removed, we slice the array starting from index 5.

# Syntax: ${array[@]:offset}
backups_to_remove=("${backups[@]:5}")

If the array contains 8 items, this slice extracts items from index 5, 6, and 7, leaving indices 0 through 4 untouched.

4. Safe Execution: Avoiding the "Angry Manager" Scenario

DevOps Best Practice: Never pipe a list of files directly into a destructive command like rm -rf without verifying the targets first. Accidental deletions in production can cause catastrophic data loss and downtime.

Before executing the deletion loop, always print the target files to the console or log files to confirm that only the correct, outdated backups are targeted.

# Safe verification step
echo "Backups identified for removal: ${backups_to_remove[@]}"

Once verified, we can safely iterate through the array and delete each file:

for backup in "${backups_to_remove[@]}"; do
    rm -f "$backup"
    echo "Deleted old backup: $backup"
done

5. Complete Backup Rotation Script

Below is the complete, production-ready Bash script implementing the safe backup rotation logic.

#!/bin/bash

# Define backup directory
BACKUP_DIR="/home/ubuntu/backups"

# 1. Retrieve backups sorted by modification time (newest first)
# Suppress errors if the directory is empty or inaccessible
backups=($(ls -dt $BACKUP_DIR/backup_* 2>/dev/null))

# 2. Check if backup count exceeds the threshold of 5
if [ "${#backups[@]}" -gt 5 ]; then
    echo "Current backup count: ${#backups[@]}. Threshold exceeded. Performing rotation..."

    # 3. Slice array to get backups starting from index 5 (oldest ones)
    backups_to_remove=("${backups[@]:5}")

    echo "The following old backups will be removed:"
    for file in "${backups_to_remove[@]}"; do
        echo " - $file"
    done

    # 4. Perform safe deletion
    for file in "${backups_to_remove[@]}"; do
        rm -rf "$file"
        echo "Successfully removed: $file"
    done
else
    echo "Current backup count: ${#backups[@]}. No rotation required."
fi

6. Running Scripts in the Background (nohup)

When running long-running scripts (like large database backups or migrations) over an SSH session, terminating the session or losing network connectivity will kill the parent shell process, terminating your script mid-execution.

To prevent this, run the script in the background using nohup (No Hang Up).

The nohup Workflow

Command Syntax

nohup ./backup.sh > backup.log 2>&1 &

• nohup: Prevents the process from being killed when the terminal session closes.
• ./backup.sh: The target script to execute.
• > backup.log: Redirects standard output (stdout) to a log file.
• 2>&1: Redirects standard error (stderr / fd 2) to the same location as standard output (fd 1).
• &: Runs the command in the background, freeing up the terminal prompt immediately.

7. Managing Scripts as System Services (systemctl)

For production-grade deployments, scripts should be managed by the system's service manager, systemd. This ensures that scripts start automatically on system boot, restart automatically if they crash, and can be monitored using standard system tools.

Step 7.1: Create a systemd Service File

Create a service configuration file under /etc/systemd/system/:

sudo nano /etc/systemd/system/backup.service

Add the following configuration:

[Unit]
Description=Automated Backup Rotation Service
After=network.target

[Service]
Type=simple
ExecStart=/home/ubuntu/scripts/backup.sh
Restart=on-failure
User=ubuntu

[Install]
WantedBy=multi-user.target

Step 7.2: Manage the Service with systemctl

Once the service file is created, use the following commands to manage it:

# Reload systemd to recognize the new service file
sudo systemctl daemon-reload

# Start the service immediately
sudo systemctl start backup.service

# Enable the service to run automatically on system boot
sudo systemctl enable backup.service

# Check the real-time status of the service
sudo systemctl status backup.service

Summary and Key Takeaways

• Standard Error Redirection (2>/dev/null): Use this to discard unwanted error messages and keep your automation logs clean.
• Array Length in Bash: Use ${#array_name[@]} to determine the number of elements in an array.
• Array Slicing Syntax: Use ${array[@]:offset} to slice elements starting from a specific index. This is highly effective for retention policies.
• Safe Deletion Practices: Always print and verify the list of files targeted for deletion (backups_to_remove) before executing destructive commands like rm -rf.
• Background Execution (nohup): Use nohup combined with & to run long-running scripts in the background, ensuring they survive terminal disconnections.
• Systemd Integration (systemctl): For production environments, wrap your automation scripts in systemd service units to enable auto-restarts, boot-time execution, and centralized logging.