Day 13 | Python for DevOps: Boto3 Beginner to Advanced Guide with Projects

This chapter covers Boto3, the official AWS SDK for Python, and demonstrates how devops engineers leverage it to programmatically manage cloud infrastructure. We will explore the core differences between scripting and templating tools, transition from basic Boto3 operations to serverless computing with AWS Lambda, and build a production-grade Cloud Cost Optimization project that automatically detects and purges stale EBS snapshots.

1. Why Do We Need Programmatic AWS Interaction?

When managing cloud infrastructure, manual operations via the AWS Management Console (UI) quickly become unsustainable. For repetitive tasks—such as spinning up dozens of EC2 instances, auditing S3 buckets, or cleaning up unused resources—manual intervention is slow, error-prone, and impossible to scale.

To automate these tasks, AWS exposes its services via REST APIs. While you can interact with these APIs using raw HTTP requests (e.g., via Python's requests module), doing so requires manually handling complex authentication signatures, request headers, pagination, and error codes.

Boto3 is the official AWS Software Development Kit (SDK) for Python. It abstracts these low-level API complexities into clean, idiomatic Python classes and methods, allowing you to write robust automation scripts in a fraction of the time.

2. The Core Mental Model: Scripting vs. Templating

DevOps engineers have access to multiple tools for managing AWS resources: the AWS CLI, Boto3, Terraform, and CloudFormation. These tools fall into two distinct operational categories: Scripting and Templating.

When to Use Scripting (Boto3) over Templating (Terraform)

While Terraform is ideal for provisioning static infrastructure (e.g., VPCs, databases, and subnets), it is poorly suited for dynamic, short-lived, or event-driven tasks.

For example, if you need to run a script every morning at 10:00 AM to scan for unattached EBS volumes and delete them, setting up a Terraform state file, writing HCL templates, and running terraform apply is highly inefficient. A lightweight Python script running inside an ephemeral AWS Lambda function is the industry-standard approach for these operational tasks.

3. Prerequisites & Environment Setup

To successfully automate AWS with Boto3, you must first understand how to perform the operations manually.

Rule of Automation: You cannot automate what you do not understand manually. If you do not know what an AMI ID, instance type, or key pair is in the AWS UI, you will struggle to configure them in Boto3 or Terraform.

Step 1: Install the AWS CLI and Authenticate

To run Boto3 scripts locally or in a development container (such as GitHub Codespaces or VS Code), you must authenticate your environment with AWS.

1. Open your terminal and run the configuration command:
   BASH

   Copy

   aws configure
   

2. Provide your credentials when prompted:
   • AWS Access Key ID: YOUR_ACCESS_KEY_ID
   • AWS Secret Access Key: YOUR_SECRET_ACCESS_KEY
   • Default region name: us-east-1 (or your preferred region)
   • Default output format: json

Note: You can generate these credentials in the AWS Console by navigating to IAM -> Users -> [Your User] -> Security Credentials -> Create Access Key.

Step 2: Install Boto3

Install the Boto3 library using pip:

pip install boto3

(If you encounter environment conflicts, try pip3 install boto3).

4. Basic Boto3 Mechanics: Clients vs. Resources

Boto3 provides two distinct interfaces for interacting with AWS APIs: Clients and Resources.

1. Client (Low-Level): Maps directly to the underlying AWS HTTP/JSON APIs. It returns raw dictionary responses and requires you to handle pagination and low-level parameters manually.
2. Resource (High-Level): Offers an object-oriented abstraction. It hides the underlying API calls and represents AWS resources as Python objects.

Critical Architectural Note: The Boto3 maintainers have announced that Resources are frozen. No new AWS services or features will receive Resource-level support. Therefore, production-grade DevOps scripts should strictly use Clients (boto3.client) to ensure future-proof compatibility.

Code Example 1: Creating an S3 Bucket (Client-Style)

Create a file named create_bucket.py:

import boto3

# Initialize the low-level S3 client
s3_client = boto3.client('s3')

# Define a globally unique bucket name
bucket_name = "abi-demo-boto3-youtube-123"

try:
    # Create the S3 bucket
    response = s3_client.create_bucket(
        Bucket=bucket_name
    )
    print(f"Bucket '{bucket_name}' created successfully!")
    print(response)
except Exception as e:
    print(f"Error creating bucket: {e}")

Code Example 2: Retrieving S3 Bucket Access Control Lists (ACL)

Once a resource is created, Boto3 returns responses as Python dictionaries (parsed from raw JSON). You can parse these dictionaries to extract specific metadata.

import boto3
import pprint

s3_client = boto3.client('s3')
bucket_name = "abi-demo-boto3-youtube-123"

try:
    # Retrieve the ACL configuration
    response = s3_client.get_bucket_acl(Bucket=bucket_name)
    
    # Pretty print the raw dictionary response
    pprint.pprint(response)
    
    # Extract specific fields
    owner = response['Owner']['DisplayName']
    print(f"\nBucket Owner: {owner}")
    
except Exception as e:
    print(f"Error retrieving ACL: {e}")

5. Serverless Architecture: EC2 vs. AWS Lambda

When executing automation scripts, running them on a persistent virtual machine like EC2 is highly inefficient. Instead, DevOps engineers use AWS Lambda, a serverless compute service.

Real-Life Analogy: The Food Delivery Workflow

Think of an EC2 instance as a dedicated restaurant kitchen that you rent 24/7. Whether customers are ordering food or the kitchen is completely empty, you pay the rent, electricity, and staff costs continuously.

AWS Lambda is like a ghost kitchen service that only charges you for the exact seconds a chef is preparing a specific order.

• When a customer places an order (an Event), a chef instantly appears, cooks the meal (executes the code), and immediately leaves.
• The kitchen is torn down the moment the order is complete. You pay only for the compute milliseconds used during cooking.

Architectural Comparison

6. DevOps Use Cases for AWS Lambda

AWS Lambda is the backbone of modern cloud operations. Its event-driven nature makes it ideal for several key DevOps patterns:

• Cloud Cost Optimization: Automatically identifying and deleting stale resources (e.g., unattached EBS volumes, old snapshots, idle load balancers, or empty S3 buckets).
• Security & Compliance Auditing:
  • Scanning for S3 buckets that have been accidentally made public and immediately reverting their permissions.
  • Detecting when a developer creates an outdated gp2 EBS volume and automatically upgrading it to a high-performance, cost-effective gp3 volume.
• Automated Backups & Maintenance: Triggering automated database snapshots or cleaning up expired application logs on a cron schedule.

7. Step-by-Step Evolution of the Cost Optimization Project

The Problem: Stale EBS Snapshots

Developers frequently spin up EC2 instances with attached Elastic Block Store (EBS) volumes. To safeguard their work, they configure daily backups by taking EBS Snapshots.

When the project ends, the developer terminates the EC2 instance. While the EC2 instance and its root EBS volume are deleted, the daily EBS Snapshots remain in storage indefinitely. Because these snapshots are no longer associated with an active volume, they become stale and incur unnecessary monthly storage costs.

The Automated Solution Workflow

8. Writing the Lambda Function Code

Below is the complete, production-grade Python script designed to run inside AWS Lambda. It uses Boto3 to scan all EBS snapshots, checks if their parent volumes exist, verifies if those volumes are attached to active EC2 instances, and deletes them if they are stale.

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    # Step 1: Get all EBS snapshots owned by this AWS account
    # 'self' filters out public snapshots shared by other accounts
    snapshots_response = ec2.describe_snapshots(OwnerIds=['self'])
    
    # Step 2: Get all active EC2 instances (to verify attachments)
    instances_response = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    )
    
    # Extract running instance IDs into a set for O(1) lookup complexity
    active_instance_ids = set()
    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])

    # Step 3: Get all active volumes
    volumes_response = ec2.describe_volumes()
    active_volume_ids = {volume['VolumeId'] for volume in volumes_response['Volumes']}

    # Step 4: Iterate through snapshots and evaluate their status
    for snapshot in snapshots_response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')

        # Scenario A: The volume associated with the snapshot no longer exists
        if not volume_id or volume_id not in active_volume_ids:
            try:
                ec2.delete_snapshot(SnapshotId=snapshot_id)
                print(f"Deleted stale snapshot {snapshot_id} (Associated volume {volume_id} does not exist).")
            except Exception as e:
                print(f"Failed to delete snapshot {snapshot_id}: {str(e)}")
                
        # Scenario B: The volume exists, but it is not attached to any running EC2 instance
        else:
            try:
                volume_info = ec2.describe_volumes(VolumeIds=[volume_id])['Volumes'][0]
                attachments = volume_info.get('Attachments', [])
                
                # If the volume has no attachments, or is attached to an inactive instance
                is_attached_to_active = any(att['InstanceId'] in active_instance_ids for att in attachments)
                
                if not is_attached_to_active:
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted stale snapshot {snapshot_id} (Associated volume {volume_id} is not attached to any running instance).")
            except Exception as e:
                # Handle edge cases where the volume might be deleted mid-execution
                print(f"Skipping snapshot {snapshot_id} due to error: {str(e)}")

Algorithmic Complexity Analysis

The script performs lookups across snapshots, volumes, and instances. By converting active instance IDs and volume IDs into Python sets, we optimize lookup times:

• Instance ID Lookup Complexity: $\mathcal{O}(1)$
• Volume ID Lookup Complexity: $\mathcal{O}(1)$
• Total Time Complexity: $\mathcal{O}(N)$ where $N$ is the total number of snapshots owned by the account. This ensures the Lambda function executes well within the default timeout limits even for large-scale AWS environments.

9. IAM Permissions & Least Privilege Configuration

By default, an AWS Lambda function runs with a basic execution role that only allows it to write logs to Amazon CloudWatch. It does not have permission to view or delete EC2 resources.

To adhere to the Principle of Least Privilege, we must create a custom IAM policy that grants only the exact API permissions required by our script.

Step 1: Create the Custom IAM Policy

1. Navigate to the IAM Console -> Policies -> Create Policy.
2. Switch to the JSON editor tab and paste the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeSnapshots",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        }
    ]
}

3. Name the policy CostOptimizationEBSPolicy and click Create Policy.

Step 2: Attach the Policy to the Lambda Execution Role

1. Open your Lambda function in the AWS Console.
2. Go to the Configuration tab -> Permissions.
3. Click on the Role Name link to open the execution role in the IAM console.
4. Click Add Permissions -> Attach Policies.
5. Search for CostOptimizationEBSPolicy, select it, and click Attach Policy.

Step 3: Adjust Lambda Timeout Settings

The default execution timeout for an AWS Lambda function is 3 seconds. Because our script queries multiple AWS APIs (which can take several seconds to respond), a 3-second timeout may cause the function to terminate prematurely.

1. In your Lambda function, navigate to Configuration -> General configuration -> Edit.
2. Increase the Timeout to 10 seconds (or more, depending on your environment's scale).
3. Click Save.

10. Event-Driven Automation with CloudWatch / EventBridge

To make this cost optimization process fully automated, we can schedule our Lambda function to run periodically using Amazon EventBridge (CloudWatch Events).

Configuring the Schedule

1. Navigate to the Amazon EventBridge console.
2. Click Schedules -> Create Schedule.
3. Provide a name (e.g., WeeklyEBSSnapshotCleanup).
4. Under Schedule pattern, select Recurring schedule.
5. Choose Cron-based schedule and enter your desired execution interval. For example, to run the cleanup every Monday morning at 8:00 AM UTC, use:
   CRON

   Copy

   0 8 ? * MON *
   

6. Under Target, select AWS Lambda and choose your cost-optimization-ebs-snapshot function.
7. Complete the wizard to enable the schedule. EventBridge will now automatically trigger your Boto3 cleanup script on the specified schedule.

Summary and Key Takeaways

• Boto3 is the official Python SDK for AWS, enabling programmatic resource management by abstracting low-level REST API calls.
• Clients vs. Resources: Always use boto3.client for production code. The high-level boto3.resource interface is frozen and will not support future AWS services.
• Serverless Compute: AWS Lambda provides ephemeral, auto-scaling compute that charges only for active execution milliseconds, making it highly cost-efficient for automation scripts compared to persistent EC2 instances.
• Event-Driven Operations: Lambda functions are triggered by events (e.g., S3 uploads, API calls, or EventBridge cron schedules), eliminating the need for manual execution.
• Stale Resource Cleanup: A primary responsibility of DevOps engineers is cloud cost optimization. Automatically identifying and purging unattached EBS volumes and orphaned snapshots can save organizations thousands of dollars.
• Least Privilege Security: Never grant wildcard (*) admin permissions to automation scripts. Always construct custom IAM policies containing only the specific API actions needed (e.g., ec2:DescribeSnapshots, ec2:DeleteSnapshot).
• Execution Timeouts: Always adjust the default 3-second Lambda timeout to accommodate API latency when querying multiple cloud resources.