Terraform + Ansible End-to-End Demo: Real-Time Infrastructure Orchestration with Spacelift

Provisioning infrastructure and configuring that infrastructure are two fundamental pillars of DevOps engineering. While tools like Terraform excel at infrastructure provisioning and Ansible dominates configuration management, using either tool in isolation for both tasks leads to severe operational scaling limitations. This chapter explores how to build a robust, enterprise-grade infrastructure pipeline by combining Terraform and Ansible using Spacelift as an intelligent orchestration layer.

1. Why Do We Need an Infrastructure Orchestrator?

In modern cloud environments, relying on a single tool for both provisioning and configuration management creates significant friction:

• Terraform Provisioners Limitations: While Terraform can invoke configuration scripts via provisioners, doing so at scale lacks reliability, proper error handling, state synchronization, and robust retry mechanisms.
• Ansible Provisioning Limitations: While Ansible is exceptionally powerful for configuration management, using it to spin up raw cloud infrastructure lacks the maturity, state tracking, and dependency graphing found in Terraform.
• The Hybrid Dilemma: Organizations frequently adopt Terraform for infrastructure and Ansible for configuration, necessitating an orchestrator to bridge the gap cleanly.

Core Responsibilities of an Infrastructure Orchestrator

1. Order of Execution: Guaranteeing that configuration management runs only after underlying cloud resources are fully provisioned and their outputs (such as IP addresses) are available.
2. Policy Management & Compliance: Enforcing governance rules across teams (e.g., preventing the creation of EC2 instances with open inbound traffic or non-compliant security groups).
3. Strict RBAC (Role-Based Access Control): Restricting powerful cloud credentials and deployment pipelines so that only authorized engineers can trigger modifications.
4. Tool Version Management: Automatically installing, tracking, and upgrading underlying toolsets (Terraform, Ansible, OpenTofu, Pulumi) without manual developer intervention.

2. Core Mental Model & Real-World Analogy

Real-Life Analogy: Imagine building a house. Terraform acts as the general contractor who pours the concrete foundation, erects the framing, and installs the doors and windows (provisioning the physical structure). Ansible acts as the interior decorator who paints the walls, installs appliances, and hooks up the electronics (configuring the environment).

Without an orchestrator (the project manager), the interior decorator might show up before the roof is built, leading to absolute chaos. Spacelift is the master project manager who ensures the foundation is completely finished, inspects the blueprints against building codes (policy management), and hands the keys to the right team members (RBAC).

3. The Naive Approach vs. The Enterprise Orchestrated Approach

Without a dedicated orchestrator like Spacelift, teams are forced to build and maintain complex internal custom tooling. The comparison below highlights the operational contrast:

4. Spacelift Architecture & Primary Building Blocks

Before building stacks, it is essential to understand the core architectural components of Spacelift:

• Spaces: Isolated hierarchical workspaces (containers) used to segregate environments (e.g., Root Space -> Dev Console Space) or business units.
• Stacks: The primary operational building block. A stack binds a version control system (VCS) repository, a specific subfolder path, and an execution tool (e.g., Terraform or Ansible) together.
• Contexts: Reusable configuration bundles containing shared environment variables, secrets, files, or SSH keys that can be attached across multiple stacks.
• Cloud Integrations: Secure connections linking Spacelift spaces to public cloud providers (AWS, Azure, GCP) using IAM roles and trust policies.
• Policies: Governance rules enforcing compliance, cost controls, and security standards before any infrastructure changes are applied.

5. Step-by-Step Implementation: Repository Layout

For this demo, a GitHub repository (spacelift-demo) is pre-populated with two distinct directory structures:

6. Configuring Shared Contexts & AWS Cloud Integration

Step 6.1: Creating Shared SSH Keys in Context

To allow Ansible to communicate with the newly provisioned EC2 instances, both Terraform and Ansible must share access to cryptographic keys.

1. Generate standard SSH keys locally:
   BASH

   Copy

   ssh-keygen -t rsa -b 2048 -f id_rsa
   

2. In Spacelift, navigate to Contexts and create a new context named ansible-key.
3. Mount the public key (id_rsa.pub) for Terraform deployment and the private key (id_rsa) for Ansible connection within this context.

Step 6.2: Establishing AWS Cloud Integration

To allow Spacelift to provision resources in AWS without hardcoding secret keys:

1. In the AWS IAM Console, choose Create Role -> Custom trust policy.
2. Paste the custom trust policy provided by Spacelift under your Cloud Integration settings:
   JSON

   Copy

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/spacelift.io"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "spacelift.io:sub": "stack:dev-console:*"
           }
         }
       }
     ]
   }
   

3. Attach administrator permissions (or scoped least-privilege policies) to the role and name it dev-console.
4. Copy the resulting IAM Role ARN and paste it into Spacelift's Cloud Integration setup for the dev-console space to verify successful connection.

7. Creating and Verifying the Terraform Stack

1. Create a new Spacelift Space called dev-console under the root parent space.
2. Create a new Stack named terraform-stack:
   • VCS Repository: spacelift-demo
   • Project Root: TF
   • Tool: OpenTofu / Terraform
   • Attached Context: ansible-key
   • Cloud Integration: Attach the dev-console AWS integration with read/write access.
3. Trigger a manual run in Spacelift. The platform automatically initializes the working directory, downloads the AWS provider, computes the execution plan, and provisions 3 EC2 instances.

8. Creating and Configuring the Ansible Stack

Because Ansible execution depends entirely on dynamic IP addresses generated by Terraform, we configure an Ansible stack with pre-execution hooks.

1. Create a second Stack named ansible-stack inside the dev-console space:
   • VCS Repository: spacelift-demo
   • Project Root: ansible
   • Tool: Ansible
   • Playbook: install-htop.yaml
   • Context: ansible-key
   • Cloud Integration: Attach dev-console.

Setting Up Ansible Environment Variables

Navigate to the Ansible stack's environment settings and configure the required variables:

Configuring Pre-Execution Hooks

To bridge Terraform outputs into Ansible, define a Before Hook in the Ansible stack. This hook executes shell commands prior to playbook execution to generate the dynamic inventory file:

# 1. Secure private key permissions
chmod 600 /mnt/workspace/id_rsa

# 2. Initialize inventory file
echo "[all]" > /mnt/workspace/inventory.ini

# 3. Loop over Terraform stack outputs and append public IPs
for ip in $TF_VAR_instances_ip; do
  echo "$ip ansible_ssh_private_key_file=/mnt/workspace/id_rsa" >> /mnt/workspace/inventory.ini
done

9. Establishing Inter-Stack Dependencies

To automate the orchestration workflow so that Ansible runs immediately after Terraform successfully completes, establish an explicit stack dependency:

1. Navigate to terraform-stack -> Dependencies.
2. Add a dependency on ansible-stack.
3. Configure an output reference passing the Terraform output AWS instances as an input variable (instances_ip) to the Ansible stack.

10. Continuous Orchestration in Action (Scaling Up)

To verify continuous integration behavior:

1. Modify the Terraform configuration (main.tf) in the GitHub repository to scale the infrastructure from 3 instances to 4 instances:
   HCL

   Copy

   locals {
     instance_count = 4
   }
   

2. Commit and push the changes to GitHub.
3. Trigger the terraform-stack in Spacelift.
4. Observe the automated workflow:
   • Terraform provisions the new 4th EC2 instance.
   • Spacelift automatically triggers the dependent Ansible stack.
   • The before hook regenerates inventory.ini containing all 4 IP addresses.
   • Ansible connects to the newly added instance and installs htop.
5. SSH into the instances to verify:
   BASH

   Copy

   htop
   

   (The utility is confirmed installed and running across all active nodes).

Summary and Key Takeaways

• Complementary Tooling: Terraform excels at infrastructure provisioning, while Ansible excels at configuration management; combining them requires an enterprise orchestrator.
• Role of Spacelift: Spacelift automates tool versioning, state management, dependency ordering, policy governance, and strict RBAC.
• Spaces and Contexts: Spaces provide logical isolation for environments, while Contexts enable secure sharing of environment variables, files, and SSH keys across stacks.
• Cloud Integration: AWS IAM roles combined with Spacelift OIDC trust policies eliminate the security risks of hardcoded static cloud credentials.
• Stack Dependencies: Defining explicit dependencies allows output variables (such as EC2 public IPs) to flow seamlessly from Terraform into Ansible.
• Dynamic Inventory Generation: Using pre-execution hooks (Before Hooks) allows DevOps engineers to construct dynamic Ansible inventory files on-the-fly from infrastructure outputs.
• Continuous Orchestration: Any change to infrastructure definitions in version control automatically cascades through provisioning and configuration management without manual intervention.