Debugging Scripts & Advanced AWS CLI Automation

This chapter covers the transition from intermediate shell scripting to advanced automation and debugging. It focuses on integrating shell scripts with the AWS Command Line Interface (AWS CLI v2) to automate cloud infrastructure, implementing robust error handling and state polling, and utilizing essential shell debugging utilities like set -x, bash -x, and trap.

1. Resolving Nested Git Repository Conflicts

Before diving into advanced scripting, it is crucial to address a common version control pitfall: nested Git repositories. This issue occurs when you clone a Git repository inside another active Git repository.

The Problem

If you clone a child repository (e.g., django-notes-app) inside a parent repository (e.g., shell-scripting-for-devops), Git treats the nested directory as a submodule or untracked entity, preventing proper tracking, staging, and committing of the child directory's files.

The Solution

To resolve this, remove the nested .git directory or delete the nested repository entirely and manage it cleanly.

# Navigate to the target directory
cd day3

# Remove the nested repository safely
rm -rf django-notes-app

# Check git status to verify clean state
git status

# Stage, commit, and push the clean changes
git add .
git commit -m "Cleaned up nested repository and added Day 3 code"
git push origin main

2. Shell Scripting vs. Infrastructure as Code (IaC)

As a DevOps engineer, you will often use specialized Infrastructure as Code (IaC) tools like Terraform or configuration management tools like Ansible. However, understanding how to achieve the same results using shell scripts and the AWS CLI is fundamental to mastering automation.

3. AWS CLI Integration Architecture

To automate cloud resources, your local shell must communicate securely with the AWS Cloud API. This is achieved using the AWS Command Line Interface (AWS CLI).

4. Conditional Installation of AWS CLI (Idempotency)

A production-grade shell script must be idempotent—it should produce the same result regardless of how many times it is run, without causing errors or duplicate installations.

Instead of blindly running installation commands every time the script executes, the script should first check if the aws command is already available in the system's $PATH.

Real-World Analogy

Think of this like checking your refrigerator before going grocery shopping. You only buy milk if you open the fridge and find that it is missing. Buying milk every single time without checking leads to wasted resources and clutter.

Idempotent Installation Script

#!/bin/bash

# Enable strict error handling
set -e

echo "Checking if AWS CLI is installed..."

# Check if the 'aws' command exists in the system PATH
if ! command -v aws &> /dev/null; then
    echo "AWS CLI not found. Initiating installation..."
    
    # Download AWS CLI v2 installer
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    
    # Unzip the installer package
    sudo apt-get install unzip -y
    unzip awscliv2.zip
    
    # Run the installation script
    sudo ./aws/install
    
    # Clean up installation files
    rm -rf awscliv2.zip aws
    echo "AWS CLI installed successfully."
else
    echo "AWS CLI is already installed. Version: $(aws --version)"
fi

5. Automating EC2 Instance Creation & State Polling

When you request a new EC2 instance from AWS, the API immediately returns a response containing the metadata of the requested instance. However, the physical virtual machine is not immediately ready; it transitions through a series of states:

If your script attempts to configure or SSH into the instance immediately after creation, it will fail because the instance is still in the pending state.

The Solution: Polling with a while Loop

To handle this, your script must capture the InstanceId, query the AWS API repeatedly, and pause (sleep) until the instance state transitions to running.

#!/bin/bash

# Define variables
AMI_ID="ami-0c55b159cbfafe1f0" # Replace with your region's valid AMI
INSTANCE_TYPE="t2.micro"

echo "Launching EC2 Instance..."

# Create EC2 instance and extract the Instance ID using --query
INSTANCE_ID=$(aws ec2 run-instances \
    --image-id "$AMI_ID" \
    --instance-type "$INSTANCE_TYPE" \
    --query 'Instances[0].InstanceId' \
    --output text)

echo "Instance created with ID: $INSTANCE_ID"

# Initialize state variable
INSTANCE_STATE="pending"

# Poll the instance state until it is 'running'
while [ "$INSTANCE_STATE" != "running" ]; do
    echo "Current State: '$INSTANCE_STATE'. Waiting for 5 seconds..."
    sleep 5
    
    # Query the current state of the instance
    INSTANCE_STATE=$(aws ec2 describe-instances \
        --instance-ids "$INSTANCE_ID" \
        --query 'Reservations[0].Instances[0].State.Name' \
        --output text)
done

echo "Success! Instance $INSTANCE_ID is now in 'running' state and ready for use."

6. Script Debugging Essentials (set -x, bash -x, trap)

Debugging is one of the most critical skills for a DevOps engineer. When a script fails silently or behaves unexpectedly, you can use built-in shell debugging mechanisms to trace execution and handle errors gracefully.

A. Execution Tracing (set -x and bash -x)

• set -x (xtrace): Prints each command and its expanded arguments to standard error before executing it. This allows you to see exactly what variables are resolving to in real-time.
• set +x: Disables execution tracing. You can wrap specific sections of your script with set -x and set +x to debug only the problematic parts.
• bash -x script.sh: Runs the entire script in debug mode from the command line without modifying the script's source code.

B. Signal Trapping (trap)

The trap command allows you to catch system signals and shell events (like errors or exits) and execute a cleanup function or log a message before the script terminates.

#!/bin/bash

# Enable execution tracing
set -x

# Define a cleanup function
cleanup() {
    echo "An error occurred! Cleaning up temporary files..."
    rm -f temp_config.json
}

# Trap ERR signals (fires when any command fails)
trap 'cleanup' ERR

# Example execution
echo "Generating temporary configuration..."
echo '{"status": "active"}' > temp_config.json

# This invalid command will fail, triggering the trap
invalid_command_to_trigger_error

echo "This line will not execute because of the error."

7. Complete Script Execution Flow Walkthrough

The following vertical workflow illustrates the complete execution path of an advanced, self-installing, state-polling AWS automation script:

Summary and Key Takeaways

• Avoid Nested Git Repositories: Cloning a repository inside another repository breaks Git tracking. Always clean up nested .git folders or directories before staging files.
• AWS CLI v2 is the Bridge: The AWS CLI acts as the command-line interface between your local shell scripts and the AWS Cloud API.
• Write Idempotent Scripts: Always check if a tool (like AWS CLI) is already installed before executing installation commands to prevent redundant operations and errors.
• Handle Asynchronous Cloud States: Cloud resources like EC2 instances do not spin up instantly. Use while loops to poll the resource state until it reaches the desired state (e.g., running).
• Use set -x for Visibility: Enable execution tracing to print commands and expanded variables to the console, making it easy to spot where a script is failing.
• Implement trap for Resiliency: Use the trap command to catch errors (ERR) or script exits (EXIT) to clean up temporary files, release locks, or send alerts.