AWS Cost Optimization & Event-Driven Serverless Project

1. Introduction: Why Cloud Cost Optimization Matters

When organizations migrate from on-premises data centers to the cloud, leadership typically expects two primary benefits:

1. Reduction of Infrastructure Overhead: Eliminating the heavy capital expenditure (CapEx) of purchasing physical servers and maintaining dedicated on-premises infrastructure teams.
2. Cloud Cost Optimization: Ensuring that cloud infrastructure expenses are managed efficiently so that the expected financial savings actually materialize.

Note: Simply migrating workloads to public cloud platforms like AWS does not automatically guarantee lower costs. Inefficient management can easily cause cloud bills to skyrocket past traditional data center expenses.

2. The Problem: Stale Resources in Cloud Environments

In real-world cloud environments, developers and engineers frequently provision resources for experimentation, testing, or short-term features. Over time, these resources are often abandoned or forgotten, leading to stale resources.

Real-Life Analogy: The Forgotten Storage Unit

Imagine renting a storage locker to keep furniture during a home renovation. Once the renovation finishes, you move your furniture back home, but you forget to cancel the rental agreement and empty the unit. Month after month, the storage facility continues charging your credit card for a locker that holds nothing of value. Stale cloud resources operate identically: developers spin up EC2 instances, attach volumes, take daily backups (snapshots), and later delete the instances while completely forgetting to release the associated storage volumes and snapshot chains.

Common Examples of Stale Resources

• Unattached EBS Volumes: Storage volumes left behind after an EC2 instance is terminated.
• Orphaned Snapshots: Point-in-time backups of volumes that are no longer attached to any active instance or volume.
• Forgotten S3 Buckets: Object storage containing obsolete data that continues to accumulate storage fees.
• Idle EKS Clusters: Kubernetes clusters running unused test workloads 24/7.

Manually monitoring hundreds of cloud resources to catch these stale assets is impractical. Therefore, DevOps and Cloud Engineers build automated solutions to detect and purge them.

3. Cost Optimization Strategies: Notification vs. Automation

When a DevOps engineer identifies stale resources, two primary operational approaches exist:

4. Architecture & Mental Model for the Automation Project

To implement automated cost optimization without human bottlenecks, we build an event-driven serverless architecture utilizing AWS Lambda, Boto3, and Amazon EventBridge.

High-Level Execution Flow

5. Step-by-Step Demonstration Setup

Step 1: Provision an EC2 Instance & Volume

1. Navigate to the AWS EC2 Dashboard.
2. Launch a new EC2 instance (e.g., Ubuntu, t2.micro).
3. Note that launching an EC2 instance automatically creates and attaches a root EBS volume (e.g., ending with 45e).

Step 2: Create an EBS Volume Snapshot

1. In the EC2 Dashboard, go to Snapshots and click Create Snapshot.
2. Select the volume attached to your test instance.
3. Name the snapshot (e.g., test-snapshot) and create it. This generates a point-in-time backup image of the volume data.

Step 3: Simulate Developer Deletion (Leaving Stale Snapshots Behind)

1. Terminate the EC2 instance.
2. Observe that terminating the instance automatically deletes its root volume.
3. Check the Snapshots section: notice that the snapshot remains intact, orphaned from any active volume or instance. AWS will continue billing for this snapshot indefinitely.

6. Writing the AWS Lambda Cost Optimization Function

We create an AWS Lambda function running Python to inspect EC2 snapshots, compare them against active volumes and instances, and purge orphaned snapshots.

Python Lambda Code (ebs_stale_snapshots.py)

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    
    # 1. Get all running or active EC2 instances
    instances_response = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running', 'stopped']}]
    )
    active_instance_ids = set()
    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])
            
    # 2. Get all EBS volumes
    volumes_response = ec2.describe_volumes()
    active_volume_ids = set()
    for volume in volumes_response['Volumes']:
        active_volume_ids.add(volume['VolumeId'])
        
    # 3. Get all EBS snapshots owned by the account
    snapshots_response = ec2.describe_snapshots(OwnerIds=['self'])
    
    # 4. Iterate and evaluate each snapshot
    for snapshot in snapshots_response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')
        
        # Case A: Snapshot is not associated with any volume ID
        if not volume_id:
            print(f"Deleting orphaned snapshot {snapshot_id} (No volume reference found)")
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            continue
            
        # Case B: Volume exists, check if it's attached to an active instance
        try:
            vol_response = ec2.describe_volumes(VolumeIds=[volume_id])
            attachments = vol_response['Volumes'][0].get('Attachments', [])
            
            # If volume has no attachments or is attached to a non-existent instance
            is_attached_to_active = any(att['InstanceId'] in active_instance_ids for att in attachments)
            
            if not attachments or not is_attached_to_active:
                print(f"Deleting stale snapshot {snapshot_id} associated with unattached/stale volume {volume_id}")
                ec2.delete_snapshot(SnapshotId=snapshot_id)
                
        except Exception as e:
            # If the volume itself has been completely deleted, delete the snapshot
            if 'InvalidVolume.NotFound' in str(e):
                print(f"Deleting stale snapshot {snapshot_id} as parent volume {volume_id} was deleted.")
                ec2.delete_snapshot(SnapshotId=snapshot_id)
            else:
                print(f"Error processing snapshot {snapshot_id}: {str(e)}")

7. Configuring IAM Permissions & Lambda Settings

Running Lambda functions that interact with AWS APIs requires configuring execution timeouts and IAM security roles adhering to the principle of least privilege.

Lambda Configuration Adjustments

• Execution Timeout: By default, AWS Lambda functions time out after 3 seconds. Complex API calls querying multiple EC2 resources require increasing the timeout to 10 seconds.

  Note: Always keep the timeout as short as necessary to minimize AWS execution cost billing.

Creating a Custom IAM Policy (cost-optimization-ebs-policy)

To grant the Lambda execution role permission to describe and delete EC2 resources, create a custom policy with the following actions:

• ec2:DescribeInstances
• ec2:DescribeVolumes
• ec2:DescribeSnapshots
• ec2:DeleteSnapshot

Attach this policy directly to the Lambda function's IAM execution role.

8. Enterprise Reality & Production Enhancements

While deleting unattached snapshots immediately works for simple cleanups, enterprise environments require additional safeguards:

• Timestamp / Buffer Period Check: Organizations rarely delete snapshots immediately upon volume deletion. Instead, they apply a time threshold (e.g., $30\text{ days}$ or $60\text{ days}$):

• Alternative Storage Tiers (Archival): Rather than permanently deleting critical backups, mature pipelines copy snapshots to cheaper storage tiers such as Amazon S3 Glacier Deep Archive using S3 Lifecycle policies.
• Dry-Run Logging: Initially configure the Lambda function in a "dry-run" mode that logs intended deletions to Amazon CloudWatch without executing delete_snapshot(), preventing accidental data loss from logic bugs.

9. Event-Driven Scheduling with Amazon EventBridge

To prevent manual execution, integrate the Lambda function with Amazon EventBridge to automate recurring cleanups:

1. Navigate to Amazon EventBridge in the AWS Console.
2. Create a new Rule with an EventBridge Scheduler schedule expression (e.g., rate of 7 days or a cron expression).
3. Target the cost optimization AWS Lambda function.
4. The system will now automatically evaluate and purge stale snapshots on a recurring schedule without human intervention.

10. Summary and Key Takeaways

• Primary Cloud Motivation: Organizations migrate to public clouds to eliminate physical data center overhead and optimize infrastructure costs.
• The Stale Resource Problem: Forgotten EC2 instances, unattached EBS volumes, and orphaned snapshots silently inflate monthly AWS bills.
• DevOps Responsibility: Cloud and DevOps engineers are responsible for implementing proactive cost-governance automation.
• Serverless Automation Stack: Combining AWS Lambda, Python (boto3), and Amazon EventBridge creates lightweight, scalable, event-driven cleanup workflows.
• Boto3 API Integration: Scripts must query describe_instances, describe_volumes, and describe_snapshots to map dependencies before executing safe deletions.
• Production Safety Measures: Always incorporate buffer time checks (e.g., 30-day retention thresholds) and least-privilege IAM policies to prevent accidental data destruction.