Day 8: Most Asked Interview Scenarios & Advanced Terraform Workflows

Welcome to the bonus Day 8 episode of the Terraform Zero to Hero series. While the initial 7-day curriculum covered fundamental and advanced Terraform concepts with hands-on demonstrations, technical interviews frequently probe scenario-based problem-solving. This chapter dives deep into two of the most critical real-world infrastructure scenarios: migrating existing cloud resources into Terraform management and handling configuration drift detection caused by manual changes.

1. Why Do We Need Scenario-Based Terraform Knowledge?

• The Theory vs. Practice Gap: Knowing how to write a basic main.tf or configure a provider is insufficient for production engineering roles. Interviewers want to know how you handle messy, pre-existing legacy infrastructure.
• The State File Bottleneck: Terraform relies entirely on its state file (terraform.tfstate) as the single source of truth. Without proper migration techniques, introducing Terraform to an existing environment breaks tracking.
• Human Error & Out-of-Band Changes: In real-world enterprises, engineers occasionally bypass automation under pressure (e.g., late-night hotfixes), creating dangerous discrepancies between cloud state and code.

2. Core Mental Model: The Infrastructure State Bridge

Terraform operates as an intermediary engine translating declarative configuration files into real cloud APIs, strictly governed by a local or remote state file.

Note: If a resource exists in AWS but has no entry in the Terraform state file, Terraform treats it as nonexistent and will attempt to recreate it (potentially causing downtime or duplicate resource errors).

3. Scenario 1: Migrating Existing Cloud Infrastructure into Terraform

The Problem Statement

Your organization has an existing AWS environment containing unmanaged resources (e.g., an EC2 instance created via CloudFormation or manually through the AWS console). Your team is transitioning entirely to Terraform, but there is no state file and no configuration code for these existing assets. Writing hundreds of resource definitions manually is error-prone and time-consuming.

Real-Life Analogy

Imagine moving into a furnished house where the previous owner left all the furniture behind, but left no inventory list. Instead of guessing the dimensions and dragging in brand-new furniture, you want a tool to scan each existing item, catalog its exact specifications, and log them into your official moving manifest.

4. Step-by-Step Implementation of Resource Import

Step 1: Initialize the Working Directory

Navigate to your project workspace (e.g., day8/scenario1) and initialize Terraform:

terraform init

Step 2: Utilize the Modern import Block (Terraform 1.5+)

Since Terraform 1.5, native import blocks allow you to declare resources you wish to ingest. Create a main.tf file specifying your provider and an import block mapping the cloud resource ID to a target Terraform resource:

provider "aws" {
  region = "us-east-1"
}

import {
  to = aws_instance.example
  id = "i-0123456789abcdef0" # Replace with your actual AWS Instance ID
}

Step 3: Automatically Generate Configuration with terraform plan

Run the plan command with the generation flag to let Terraform automatically construct the matching resource block based on the cloud provider's current state:

terraform plan -generate-config-out=generated_resource.tf

• What happens behind the scenes: Terraform queries AWS, inspects the instance configuration (including required and optional fields such as AMI, instance type, networking, and security settings), and writes the complete configuration into generated_resource.tf.

Step 4: Clean and Integrate the Generated Code

Open generated_resource.tf and review the generated configuration. You will notice it includes both required attributes and optional default attributes fetched from AWS.

# generated_resource.tf (Sample output generated by Terraform)
resource "aws_instance" "example" {
  ami                         = "ami-0c55b159cbfafe1f0"
  instance_type               = "t2.micro"
  subnet_id                   = "subnet-0bb1c79de37d9ebef"
  vpc_security_group_ids      = ["sg-089123456789abcdef"]
  # ... optional network, ipv6, and monitoring attributes ...
}

Copy this resource block into your primary main.tf file, delete the temporary generated_resource.tf file, and remove the import block from main.tf since we are moving to the final linking stage.

Step 5: Bind the Resource to the State File

Even though the configuration exists in your .tf files, Terraform still doesn't track it until the state file is populated. Execute the explicit import command:

terraform import aws_instance.example i-0123456789abcdef0

Step 6: Verify with terraform plan

Run a plan to confirm that Terraform successfully recognizes the imported resource:

terraform plan

Expected Output:

No changes. Your infrastructure matches the configuration.

If this message appears, the migration and import were 100% successful.

5. Scenario 2: Drift Detection (Handling Manual Out-of-Band Changes)

The Problem Statement

Your infrastructure is fully managed by Terraform, and everything runs smoothly for months. However, late on a Friday shift, an engineer with direct AWS console access manually modifies an S3 bucket lifecycle rule to resolve an urgent client issue. They log out and forget to update the Terraform code.

The Core Risk

Terraform remains completely unaware of this manual modification because the state file still reflects the old configuration. Unless someone runs an explicit check, a silent configuration drift persists in production.

6. Comparing Drift Detection Strategies

When interviewers ask, "How do you handle drift detection?", proposing a single tool is insufficient. You should compare native commands versus architectural safeguards:

7. Deep-Dive: Designing the Audit Log & Lambda Notification Pipeline

To impress an interviewer with enterprise-grade system design for Scenario 2, explain the event-driven architecture used to catch out-of-band human modifications:

Architectural Breakdown:

1. CloudTrail Capture: Every API call made inside the AWS account (whether via Terraform automation or human AWS console clicks) is logged in real-time to AWS CloudTrail.
2. Event Filtering: CloudWatch watches CloudTrail logs for mutating events (e.g., UpdateBucketLifecycle, ModifyInstanceAttribute).
3. Lambda Validation: A custom AWS Lambda function parses the event payload, checking the userIdentity.principalId or ARN.
   • If the ARN matches the Terraform IAM Execution Role, the change is authorized and ignored.
   • If the ARN matches a Human IAM User (e.g., john.doe), the Lambda function flags it as an unapproved manual drift.
4. Immediate Alerting: The Lambda function publishes an alert to an SNS topic or Slack webhook, notifying the DevOps team immediately of unauthorized configuration drift.

8. Summary and Key Takeaways

• The Power of State: The Terraform state file (terraform.tfstate) is the absolute core and brain of Terraform; without it, Terraform cannot track, update, or destroy resources.
• Automated Migration: Use terraform plan -generate-config-out=<filename>.tf combined with the import block introduced in Terraform 1.5+ to rapidly ingest legacy cloud infrastructure.
• Finalizing Imports: Always pair configuration generation with the explicit terraform import <resource> <id> command to properly bind existing cloud assets to your state file.
• Verifying Integrity: Always run terraform plan post-import to verify that the output yields No changes. Your infrastructure matches the configuration.
• Configuration Drift: Drift occurs when out-of-band manual changes are made directly in the cloud provider console, creating a discrepancy with Terraform code.
• Refresh Limitations: While terraform refresh updates state files against live cloud environments, it is increasingly deprecated and requires external cron orchestration.
• Enterprise Defense: Secure production environments by enforcing strict IAM boundaries and building automated AWS CloudTrail + Lambda notification pipelines to instantly flag manual human changes.