Terraform + Ansible End-to-End Orchestration | Real-World Infrastructure Automation with Spacelift

Modern cloud engineering demands a clear separation of concerns between infrastructure provisioning and configuration management. While Terraform excels at creating and managing cloud resources, and Ansible is highly optimized for configuring those resources, combining them at enterprise scale introduces significant orchestration challenges. This guide explores how to seamlessly orchestrate Terraform and Ansible using Spacelift, establishing automated dependencies, dynamic inventory generation, and secure policy enforcement.

1. The Infrastructure Orchestration Dilemma

In DevOps engineering, provisioning infrastructure and configuring that infrastructure are two distinct phases of the deployment lifecycle.

Organizations often make the mistake of trying to force one tool to handle both phases:

• The Terraform-First Pitfall: Terraform can technically configure resources using local-exec or remote-exec provisioners. However, at scale, these provisioners are unreliable, lack robust error handling, do not maintain configuration state, and break the declarative nature of Terraform.
• The Ansible-First Pitfall: Ansible can provision cloud resources (such as EC2 instances) using cloud modules. However, Ansible is not a state-aware tool. It lacks the advanced dependency mapping, state file management, and dry-run (plan) capabilities of Terraform, making it immature for complex infrastructure provisioning.

The Solution: Hybrid Orchestration

The industry standard is to combine both tools: use Terraform to provision the raw infrastructure and Ansible to configure it. However, executing this hybrid workflow manually or via basic CI/CD pipelines introduces a new set of challenges: order of execution, dynamic inventory management, and secure state sharing.

2. The "Internal Tooling Nightmare" Analogy

Real-Life Analogy

Imagine you are tasked with building a custom internal platform to coordinate Terraform and Ansible without an external orchestrator. To make this work at scale, your internal tool must handle:

1. Tooling & Version Control: Installing, updating, and maintaining consistent versions of both Terraform and Ansible binaries across all runner environments.
2. State & Order of Execution: Ensuring Ansible only runs after Terraform has successfully completed provisioning. If Terraform fails or is delayed, Ansible must wait.
3. Dynamic Inventory Mapping: When Terraform provisions three new EC2 instances, their public IP addresses are dynamically assigned by the cloud provider. Your custom tool must capture these dynamic IPs from Terraform's output, format them, and write them into an Ansible inventory.ini file in real-time.
4. Secrets & Cloud Integration: Securely distributing AWS/Azure credentials and SSH private keys to both tools without exposing them to unauthorized users.
5. Governance & RBAC: Restricting who can run which playbooks and enforcing compliance policies (e.g., preventing the creation of EC2 instances with wide-open security groups).

Building and maintaining this custom orchestrator is a massive, error-prone engineering effort. This is where dedicated infrastructure orchestrators like Spacelift become essential.

3. Naive Manual Integration vs. Orchestrated Approach

To understand the value of a dedicated orchestrator, we can compare the manual, script-heavy approach to an automated orchestrator workflow.

4. Spacelift Building Blocks & Architecture

Spacelift acts as an orchestration engine that sits on top of your Version Control System (VCS) and coordinates various Infrastructure as Code (IaC) and configuration management tools.

Key Components

• Stack: The primary building block. A stack binds a specific VCS repository (and subdirectory) to an IaC/configuration tool (Terraform, Ansible, Pulumi, CloudFormation, or Kubernetes) and defines its execution environment.
• Space: A logical container or workspace used to isolate resources, stacks, and configurations. Spaces support hierarchical inheritance (e.g., a root space with child spaces like dev and prod).
• Context: A collection of environment variables, configuration files, or secure keys that can be attached to multiple stacks. This allows you to define a resource once (such as an SSH key pair) and share it across both Terraform and Ansible stacks.
• Cloud Integration: A secure, keyless integration mechanism (typically using OpenID Connect / OIDC) that allows Spacelift stacks to authenticate with cloud providers without storing long-lived access keys.

5. Step-by-Step Implementation: Setting Up the Foundation

This section details how to configure the foundational components in Spacelift: Spaces and Contexts.

Step 1: Create a Dedicated Space

To isolate our development resources, we create a child space under the default root space.

1. Navigate to Spaces in the Spacelift UI.
2. Click Create Space.
3. Configure the following:
   • Name: dev-console
   • Parent Space: root
   • Inherit entities: Enabled (allows child space to access global configurations if needed).

Step 2: Generate and Configure the SSH Key Context

To allow Ansible to configure the EC2 instances provisioned by Terraform, both tools must share an SSH key pair. Terraform mounts the public key onto the instances, and Ansible uses the private key to authenticate.

First, generate a standard SSH key pair locally:

ssh-keygen -t rsa -b 4096 -f id_rsa -N ""

This command generates two files:

• id_rsa (Private Key)
• id_rsa.pub (Public Key)

Next, create a Spacelift Context to securely store and distribute these keys:

1. Navigate to Contexts and click Create Context.
2. Name the context ansible-key and assign it to the dev-console space.
3. Add the keys as secure environment variables or files within the context:

Note: Storing these as files in the context ensures Spacelift mounts them directly into the workspace directory (/mnt/workspace/) during stack execution, making them easily accessible to your scripts.

6. Step-by-Step Implementation: AWS Cloud Integration

To avoid storing static AWS credentials (Access Key ID and Secret Access Key) inside Spacelift, we configure an OpenID Connect (OIDC) integration. This allows Spacelift to assume a temporary IAM role in AWS.

Spacelift Runner (OIDC Token) 
       |
       | Assumes Role via AWS STS
       v
 AWS IAM Role (dev-console) 
       |
       | Grants temporary credentials
       v
Provision/Configure AWS Resources

Step 1: Create the IAM Role in AWS

1. Log in to your AWS Console and navigate to IAM -> Roles -> Create Role.
2. Select Custom Trust Policy.
3. Copy the trust policy provided by Spacelift's AWS Integration wizard. It will look similar to this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/spacelift.io"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "spacelift.io:aud": "your-spacelift-tenant"
        },
        "StringLike": {
          "spacelift.io:sub": "space:dev-console:*"
        }
      }
    }
  ]
}

4. Attach the necessary permissions policy. For this demonstration, attach AdministratorAccess (ensure you restrict this policy in production environments).
5. Name the role dev-console-spacelift-role and save it.

Step 2: Link the Role in Spacelift

1. In Spacelift, navigate to Cloud Integrations -> AWS -> Add Integration.
2. Set the Integration Name to dev-console.
3. Select the dev-console space.
4. Paste the Role ARN of the IAM role you just created in AWS.
5. Click Save.

7. Step-by-Step Implementation: Creating the Terraform Stack

With the foundation in place, we can now configure the Terraform Stack. This stack reads our infrastructure code, assumes the AWS IAM role, and provisions the EC2 instances.

The Terraform Code

The Terraform configuration is stored in a subdirectory named /TF in our GitHub repository.

main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

locals {
  instances = {
    "instance-1" = "ubuntu"
    "instance-2" = "ubuntu"
    "instance-3" = "ubuntu"
  }
}

resource "aws_key_pair" "deployer" {
  key_name   = "deployer-key"
  public_key = file("/mnt/workspace/id_rsa.pub") # Mounted from Spacelift Context
}

resource "aws_instance" "web" {
  for_each      = local.instances
  ami           = "ami-0c7217cdde317cfec" # Ubuntu 22.04 LTS AMI
  instance_type = "t2.micro"
  key_name      = aws_key_pair.deployer.key_name

  tags = {
    Name = each.key
  }
}

outputs.tf

output "instance_ips" {
  value       = [for instance in aws_instance.web : instance.public_ip]
  description = "The public IP addresses of the provisioned EC2 instances."
}

Creating the Stack in Spacelift

1. Navigate to Stacks -> Create Stack.
2. Name: terraform-stack
3. Space: dev-console
4. Repository: Select your VCS repository.
5. Project Root: TF (relative path to the Terraform files).
6. Tool: Select Terraform (and choose your desired version, e.g., 1.5.x).
7. Contexts: Attach the ansible-key context.
8. Integrations: Attach the dev-console AWS integration with Read/Write access.
9. Click Create.

8. Step-by-Step Implementation: Creating the Ansible Stack

The Ansible stack is responsible for configuring the provisioned EC2 instances. Because Ansible requires a list of target hosts, we must dynamically generate an inventory file before the playbook runs.

The Ansible Playbook (install_htop.yml)

This playbook is stored in the /ansible directory of our repository. It targets all hosts and installs the htop system monitor.

---
- name: Configure Target EC2 Instances
  hosts: all
  become: true
  gather_facts: yes
  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Install htop package
      apt:
        name: htop
        state: present

Creating the Stack in Spacelift

1. Navigate to Stacks -> Create Stack.
2. Name: ansible-stack
3. Space: dev-console
4. Repository: Select the same VCS repository.
5. Project Root: ansible
6. Tool: Select Ansible.
7. Playbook File: install_htop.yml
8. Contexts: Attach the ansible-key context.
9. Integrations: Attach the dev-console AWS integration.

Configuring Environment Variables

To allow Ansible to run non-interactively, we must define specific environment variables in the Ansible stack's Environment tab:

Configuring the Before-Hook (Dynamic Inventory Generation)

Because the EC2 instances are created dynamically, we cannot hardcode their IP addresses. We use a Spacelift Before-Hook on the Ansible stack to generate the inventory.ini file on the fly using the outputs from the Terraform stack.

In the Ansible stack settings, navigate to Hooks -> Before Initialization and add the following script:

# 1. Set secure permissions on the private key
chmod 400 /mnt/workspace/id_rsa

# 2. Initialize an empty inventory file
echo "[all]" > /mnt/workspace/inventory.ini

# 3. Parse the JSON output passed from the Terraform stack and append IPs to the inventory
echo "$TF_OUT_instance_ips" | jq -r '.[]' >> /mnt/workspace/inventory.ini

# 4. Print the generated inventory for debugging visibility
echo "--- Generated Inventory ---"
cat /mnt/workspace/inventory.ini

9. Step-by-Step Implementation: Establishing Stack Dependencies

To tie the two stacks together into a unified pipeline, we must establish a dependency relationship. This ensures that:

1. The Ansible stack only triggers after the Terraform stack successfully completes.
2. The output variables from the Terraform stack (specifically the public IP addresses) are passed securely to the Ansible stack.

Configuring the Dependency in Spacelift

1. Navigate to the terraform-stack page.
2. Go to Settings -> Dependencies.
3. Click Add Dependency and select ansible-stack.
4. Under Output References, configure the mapping:
   • Output Name: instance_ips (the output defined in outputs.tf).
   • Input Variable Name: TF_OUT_instance_ips (the environment variable that will be injected into the Ansible runner).
5. Click Save.

10. End-to-End Execution, Scaling, and Verification

With the dependency graph established, we can trigger the entire workflow.

Step 1: Trigger the Terraform Stack

1. Navigate to the terraform-stack and click Trigger.
2. Spacelift initializes the runner, runs terraform plan, and prompts for confirmation (or applies automatically depending on your stack settings).
3. Once applied, Terraform outputs the public IP addresses:

Outputs:
instance_ips = [
  "54.210.43.12",
  "3.90.112.85",
  "18.207.220.4"
]

Step 2: Automatic Ansible Triggering

Because of the dependency rule, the successful completion of terraform-stack automatically triggers a run on ansible-stack.

During the initialization phase of the Ansible stack, the Before-Hook executes:

1. It reads the TF_OUT_instance_ips variable containing ["54.210.43.12", "3.90.112.85", "18.207.220.4"].
2. It parses this JSON array using jq and writes the IPs to /mnt/workspace/inventory.ini.
3. The Ansible execution engine starts, reads the generated inventory, connects to the instances via SSH using /mnt/workspace/id_rsa, and installs htop.

Ansible Run Log:
PLAY [Configure Target EC2 Instances] *******************************************

TASK [Gathering Facts] *********************************************************
ok: [54.210.43.12]
ok: [3.90.112.85]
ok: [18.207.220.4]

TASK [Update apt cache] ********************************************************
changed: [54.210.43.12]
changed: [3.90.112.85]
changed: [18.207.220.4]

TASK [Install htop package] ****************************************************
changed: [54.210.43.12]
changed: [3.90.112.85]
changed: [18.207.220.4]

PLAY RECAP *********************************************************************
54.210.43.12               : ok=3    changed=2    unreachable=0    failed=0
3.90.112.85                : ok=3    changed=2    unreachable=0    failed=0
18.207.220.4               : ok=3    changed=2    unreachable=0    failed=0

Step 3: Verifying Scaling Operations

To test the robustness of this orchestration, we can scale our infrastructure by adding a fourth instance.

1. Update the locals block in main.tf to include "instance-4" = "ubuntu".
2. Commit and push the change to your GitHub repository.
3. Spacelift detects the commit, triggers terraform-stack, and provisions the fourth EC2 instance.
4. Upon completion, the updated list of four IP addresses is passed to ansible-stack.
5. The Ansible stack runs, dynamically generates an inventory with all four IPs, and installs htop on the new instance while verifying the configuration on the existing three.

11. Enterprise Considerations: Policy Enforcement, RBAC, and Drift Detection

When deploying this architecture in an enterprise environment, you should leverage Spacelift's advanced governance features to ensure compliance and security.

Policy-as-Code (Rego)

Spacelift allows you to write policies using Rego (the Open Policy Agent language) to enforce compliance rules before resources are provisioned. For example, you can write a policy that blocks any Terraform plan containing an aws_security_group resource with an inbound rule of 0.0.0.0/0 (open to the world).

package spacelift

# Deny any plan that opens SSH (port 22) to the public internet
deny[msg] {
    resource := input.terraform.resource_changes[_]
    resource.type == "aws_security_group_rule"
    resource.change.after.to_port == 22
    resource.change.after.cidr_blocks[_] == "0.0.0.0/0"
    msg := sprintf("Security Group Rule for port 22 cannot be open to the public internet in resource: %v", [resource.address])
}

Role-Based Access Control (RBAC)

By using Spaces, you can restrict access to sensitive stacks. For example, you can grant developers read-only access to the prod space while giving them full read/write access to the dev-console space. This prevents unauthorized users from triggering infrastructure changes or accessing sensitive SSH keys.

Drift Detection

In enterprise environments, resources are sometimes modified directly in the cloud console, bypassing IaC. Spacelift's Drift Detection automatically runs periodic background plans. If a discrepancy is found between the actual cloud state and your Terraform state, Spacelift flags the drift and can be configured to automatically run a reconciliation pipeline to revert unauthorized changes.

12. Summary and Key Takeaways

• Separation of Concerns: Use Terraform strictly for provisioning cloud resources and Ansible strictly for configuring those resources. Avoid using Terraform provisioners or Ansible cloud modules at scale.
• Orchestration is Essential: Combining Terraform and Ansible manually requires building complex, custom internal tooling to manage versions, dependencies, and dynamic inventories.
• Spacelift Stacks: Stacks allow you to run both Terraform and Ansible in isolated, managed environments with native support for versioning and state management.
• Dynamic Inventory Generation: By using Spacelift Before-Hooks, you can dynamically generate Ansible inventory.ini files using JSON outputs passed directly from your Terraform stacks.
• Secure Context Sharing: Use Spacelift Contexts to securely share SSH key pairs and environment variables across multiple stacks without exposing secrets in your code repositories.
• Keyless Authentication: Leverage OpenID Connect (OIDC) cloud integrations to authenticate Spacelift with AWS, eliminating the need to store long-lived IAM credentials.
• Automated Dependency Pipelines: Establishing explicit stack dependencies ensures that configuration management runs automatically and reliably immediately after successful infrastructure provisioning.