Day 22: Putting It All Together: Application and Infrastructure Workflows with Terraform

The Unified Workflow

The side-by-side picture of the two pipelines, with the single point where they hand off to each other:

APPLICATION PIPELINE                    INFRASTRUCTURE PIPELINE
────────────────────────────────────    ────────────────────────────────────
Developer writes FastAPI endpoint       Engineer changes terraform-aws-asg
        │                                         │
git push → PR opens                     git push → PR opens
        │                                         │
CI: pytest (unit + integration)         CI: fmt, validate, tflint, checkov,
        │                               terraform test (unit + integration)
        │                                         │
Code review → merge to main             Code review → merge to main
        │                                         │
docker build + push to GHCR             git tag v1.5.0 → private registry
(image: sha-a1b2c3d)                    indexes new version
        │                                         │
Update image_tag var in                 Update version pin in
HCP Terraform staging workspace         environments/dev/main.tf
        │                                         │
Staging workspace: auto-apply           Dev workspace: terraform apply
        │                                         │
Smoke tests against staging ALB         Manual validation in dev
        │                                         │
Human approval gate                     Update version pin in staging
        │                                         │
Update image_tag var in prod            Staging workspace: plan review
HCP Terraform workspace                 → human approval → apply
        │                                         │
Sentinel policy check                   Update version pin in prod
        │                                         │
Human approval → prod apply             Sentinel policy check
        │                                         │
Production smoke tests                  Human approval → prod apply
        │                                         │
New endpoint live                    IMDSv2 enforced on all instances

The two pipelines are independent. They share one handoff: the infrastructure pipeline provisions and configures the cluster that the application pipeline deploys onto. The application pipeline only needs to know the ALB DNS name and the image registry URL. Everything else is the infrastructure pipeline's concern.

The practical implication: an application deployment does not require coordination with the infrastructure team, and vice versa. The FastAPI team can ship new endpoints five times a day without involving the platform team. The platform team can update the ASG configuration without blocking a FastAPI release. This is exactly what we would expect in a real working environments

Immutable Artifacts

Both pipelines share a foundational principle: build once, promote everywhere. Never rebuild the artifact for each environment — instead, build it once in CI and pass the same artifact through dev, staging, and production.

Application artifacts: Docker images

docker build -t ghcr.io/mohamednourdine/fastapi-items:sha-a1b2c3d .
docker push ghcr.io/mohamednourdine/fastapi-items:sha-a1b2c3d

# The same sha-a1b2c3d image runs in dev, staging, and production
# image_tag = "sha-a1b2c3d" is the variable that flows through all three environments

The image tagged with the git SHA is the immutable artifact. It does not change between environments. What changes is the Terraform variable that tells each environment which image to run.

Infrastructure artifacts: versioned modules

# terraform-aws-asg at v1.5.0 is the immutable artifact
git tag v1.5.0 && git push origin v1.5.0

# The same v1.5.0 code runs in dev, staging, and production
# version = "~> 1.5" is the version pin that each environment adopts sequentially

A module version is immutable once tagged. If a bug is found, a new tag (v1.5.1) is created — the old tag is never moved or deleted. This means the dev environment can be at v1.5.0 while prod is still at v1.4.3, with a clear upgrade path and a clean git history of what changed between them.

Why immutability matters

If you rebuild the Docker image on each deploy, two "identical" deploys can produce different results — a dependency updated on PyPI between the two builds, or a base image patch changed behavior. The same applies to Terraform modules: if the module source is a GitHub branch URL instead of a tag, terraform init on Monday and terraform init on Wednesday can pull different code.

Immutable artifacts eliminate this entire category of "worked in staging, broke in prod" incidents.

Sentinel Policies: A Deeper Look

Day 19 introduced Sentinel as the policy enforcement layer between terraform plan and terraform apply. Day 21 applied it to a single workspace. The production-scale pattern is policy sets — collections of policies applied to multiple workspaces simultaneously.

Sentinel requires the HCP Terraform Plus tier. Policy sets, Sentinel, and OPA enforcement are all gated behind the paid Plus edition — the free Standard tier does not include them. On the free tier, the equivalent guardrails come from checkov and conftest (OPA) running in CI before the plan ever reaches HCP Terraform. The patterns below still apply; only the enforcement engine differs.

Why Sentinel runs after plan, before apply. Sentinel evaluates the planned changes — the JSON output of terraform plan — not live infrastructure. This means a non-compliant or destructive change is blocked before any resource is touched. The same plan that the reviewer sees in the UI is the plan Sentinel evaluates, so policy results match what will actually happen on apply.

Policy sets

A policy set is a group of Sentinel policies stored in a VCS repository, connected to HCP Terraform, and applied to one or more workspaces (or the entire organization):

github.com/mohamednourdine/terraform-policies/
├── policies/
│   ├── rds-encryption.sentinel
│   ├── imdsv2-required.sentinel
│   ├── instance-type-allowlist.sentinel
│   └── tagging-required.sentinel
├── sentinel.hcl            ← declares which policies are in the set and their enforcement levels
└── README.md

The sentinel.hcl file connects policies to enforcement levels:

# sentinel.hcl
policy "rds-encryption" {
  source            = "./policies/rds-encryption.sentinel"
  enforcement_level = "hard-mandatory"
}

policy "imdsv2-required" {
  source            = "./policies/imdsv2-required.sentinel"
  enforcement_level = "hard-mandatory"
}

policy "instance-type-allowlist" {
  source            = "./policies/instance-type-allowlist.sentinel"
  enforcement_level = "soft-mandatory"
}

policy "tagging-required" {
  source            = "./policies/tagging-required.sentinel"
  enforcement_level = "advisory"
}

In HCP Terraform → Settings → Policy Sets → connect this repository. The policy set can be applied globally (all workspaces in the organization) or scoped to specific workspaces or projects.

Writing practical policies

Sentinel is its own policy language (HCL-flavoured); the snippets below use the tfplan/v2 import, which is the current path — the legacy tfplan import has different attribute access and should not be used in new policies.

1. RDS encryption — hard mandatory

# policies/rds-encryption.sentinel
import "tfplan/v2" as tfplan

rds_instances = filter tfplan.resource_changes as _, rc {
    rc.type is "aws_db_instance" and
    (rc.change.actions contains "create" or rc.change.actions contains "update")
}

main = rule {
    all rds_instances as _, rc {
        rc.change.after.storage_encrypted is true
    }
}

Hard mandatory — no override. An unencrypted RDS instance can never reach any workspace this policy is applied to.

2. IMDSv2 enforcement — hard mandatory

# policies/imdsv2-required.sentinel
import "tfplan/v2" as tfplan

launch_templates = filter tfplan.resource_changes as _, rc {
    rc.type is "aws_launch_template" and
    rc.change.actions contains "create"
}

main = rule {
    all launch_templates as _, rc {
        rc.change.after.metadata_options is not null and
        rc.change.after.metadata_options[0].http_tokens is "required"
    }
}

3. Instance type allowlist — soft mandatory

Prevents engineers from accidentally provisioning expensive instances in production while allowing an admin override for justified exceptions:

# policies/instance-type-allowlist.sentinel
import "tfplan/v2" as tfplan

approved_types = [
  "t3.micro", "t3.small", "t3.medium", "t3.large",
  "m5.large", "m5.xlarge",
  "c5.large", "c5.xlarge",
]

launch_templates = filter tfplan.resource_changes as _, rc {
    rc.type is "aws_launch_template" and
    rc.change.actions contains "create"
}

main = rule {
    all launch_templates as _, rc {
        rc.change.after.instance_type in approved_types
    }
}

Soft mandatory — a senior engineer can override with a written justification if a workload genuinely needs a GPU or memory-optimized instance that is not on the allowlist.

4. Required tags — advisory

# policies/tagging-required.sentinel
import "tfplan/v2" as tfplan

required_tags = ["Environment", "Project", "ManagedBy"]

# Only resource types that actually support tags. Asserting tags on
# aws_route, aws_security_group_rule, aws_iam_role_policy_attachment,
# aws_route_table_association, etc. would warn on every run.
taggable_types = [
  "aws_instance", "aws_launch_template", "aws_db_instance",
  "aws_s3_bucket", "aws_lb", "aws_lb_target_group",
  "aws_autoscaling_group", "aws_security_group", "aws_vpc",
  "aws_subnet", "aws_eip", "aws_nat_gateway",
]

taggable_resources = filter tfplan.resource_changes as _, rc {
    rc.change.actions contains "create" and
    rc.type in taggable_types
}

main = rule {
    all taggable_resources as _, rc {
        rc.change.after.tags is not null and
        all required_tags as tag {
            tag in keys(rc.change.after.tags)
        }
    }
}

Advisory — logs a warning in the run output but does not block. Useful when migrating legacy resources that do not yet have tags: the advisory surfaces the gap without stopping the deployment.

Policy sets are scoped per environment. The same imdsv2-required policy can be advisory in dev (warn so the team notices) and hard-mandatory in prod (block, no override). HCP Terraform supports separate policy sets attached to different workspaces or projects, so guardrails ratchet up as code promotes from dev → staging → prod. The dev policy set might be 4 advisory policies; the prod set is 4 hard-mandatory policies pointing at the same .sentinel files.

The policy lifecycle

Policies follow the same workflow as infrastructure code: version-controlled, code-reviewed, merged, and promoted. A policy change that accidentally makes hard-mandatory too restrictive can block all workspaces. The same discipline applies:

Change to rds-encryption.sentinel
          │
PR → CI: sentinel test (Sentinel's own test runner)
          │
Code review → merge to main
          │
HCP Terraform detects the VCS change, updates the policy set
          │
All connected workspaces now run the new policy on next plan

The HCP Terraform VCS Workflow

The VCS workflow connects every part of Days 19–22 into an automated system. Here is the complete setup for the FastAPI production workspace:

Workspace configuration

Workspace: fastapi-prod
├── VCS connection: github.com/mohamednourdine/terraform-infra
│   ├── Branch: main
│   ├── Working directory: environments/prod
│   └── Trigger pattern: environments/prod/**
│
├── Variables:
│   ├── image_tag          = "sha-a1b2c3d"    (Terraform var, updated by CI)
│   ├── environment        = "prod"             (Terraform var)
│   ├── instance_type      = "t3.medium"        (Terraform var)
│   └── min_size           = 3                  (Terraform var)
│
├── AWS credentials: Dynamic Provider Credentials (OIDC)
│   └── Role ARN configured under Settings → Dynamic Credentials,
│       not as a workspace env var. Matches the OIDC pattern from Day 16/19 —
│       no static AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY anywhere.
│
├── Auto-apply: disabled
├── Run triggers: fastapi-prod-networking, fastapi-prod-alb
│   └── Trigger queues a plan in fastapi-prod; approval is still required
│       because auto-apply is disabled. Triggers do not bypass the gate.
└── Policy sets: mohamednourdine-org/terraform-policies (Plus tier)

The complete flow from git push to applied infrastructure

1. Engineer merges PR to main in terraform-infra
         │
2. GitHub sends webhook to HCP Terraform
         │
3. HCP Terraform checks trigger pattern:
   environments/prod/** → fastapi-prod workspace triggered
         │
4. HCP Terraform queues a speculative plan for the PR (if it's still open)
   or a real plan (if it just merged)
         │
5. terraform plan executes in HCP Terraform's environment:
   - Uses workspace variables (no local credentials needed)
   - Uses module versions from private registry
   - Reads cross-workspace state via terraform_remote_state
         │
6. Plan output appears in HCP Terraform UI and as GitHub PR status check
         │
7. Sentinel policy set evaluates the plan:
   - rds-encryption:       PASS
   - imdsv2-required:      PASS
   - instance-type-allowlist: PASS (t3.medium is on allowlist)
   - tagging-required:     ADVISORY (2 resources missing Project tag)
         │
8. Run pauses — auto-apply is disabled, approval required
         │
9. Senior engineer reviews plan + Sentinel results in HCP Terraform UI
   Sees: 1 resource to update in-place, 0 to destroy, advisory on tags
   Clicks: Confirm & Apply
         │
10. terraform apply executes
         │
11. HCP Terraform sends Slack notification:
    "fastapi-prod: apply complete. 1 change applied. Run #run-abc123"
         │
12. Run logged in HCP Terraform audit trail with:
    - Who approved it (engineer name)
    - Full plan output
    - Apply duration and result
    - Which commit triggered it (SHA + branch)

This is the complete audit trail that a compliance review needs: what changed, who approved it, when, and what the resulting infrastructure state was.

Speculative plans on pull requests

Speculative plans on PRs require "Automatic speculative plans" to be enabled in the workspace's VCS settings (it is on by default for VCS-connected workspaces, but can be disabled). Without it, PRs do not get a status check.

Before any of this reaches production, speculative plans give reviewers an infrastructure preview inside the PR itself:

mohamednourdine/terraform-infra PR #47
"Update ASG module to v1.5.0 (IMDSv2 enforcement)"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HCP Terraform — fastapi-prod               Plan succeeded
Plan: 0 to add, 1 to change, 0 to destroy.
View full plan → app.terraform.io/run/run-xyz789

Changes:
  ~ aws_launch_template.web
    + metadata_options.http_tokens = "required"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The reviewer sees the exact resource change in the PR without needing to check out the branch locally or run terraform plan themselves.

What the Stack Looks Like at Day 22

The FastAPI infrastructure has grown considerably since Day 9. The comparison is useful for understanding how each day's concepts compound:

Every row maps to a specific day in the series. The final stack is not a rewrite of the Day 9 stack — it is the Day 9 stack with each subsequent day's additions layered on top. That is the point of the incremental approach.

Key Lessons from 22 Days

1. The plan is the product. So far, we have seen that the most valuable thing Terraform produces is not the infrastructure — it is the plan. The plan tells you exactly what will change before anything changes. Every workflow in this series is built around making the plan visible to the right people at the right time: in CI, as a PR comment, in the HCP Terraform UI before an approval gate.

2. State is the source of truth — treat it accordingly. The Terraform state file is more important than the configuration files. The config can be regenerated from the state; the reverse is not true. Protect state with S3 versioning, DynamoDB locking, and prevent_destroy on databases. My kind advice, never edit state directly except through terraform state commands except you realy know what you are doing.

3. Module versioning is the difference between "shared" and "copied". Shared modules without version pins are a liability — a change to the module silently affects every consumer on the next terraform init. Semantic tags (v1.5.0), pinned version constraints (~> 1.5), and a committed .terraform.lock.hcl make modules genuinely reusable across teams.

4. Infrastructure tests catch what validate cannot. terraform validate is syntax checking. tflint is schema checking. checkov is compliance checking. None of them catch "the security group blocks port 8000 so the ALB never reaches the app." Only Terratest — which deploys real resources and makes real HTTP requests — catches behavioral failures. The test pyramid is genuinely a pyramid: static analysis is free, integration tests cost $0.50, and E2E tests cost $2. Invest in all three.

5. Rollback for infrastructure is a plan, not a button. Application rollbacks are fast and reversible — swap the image tag. Infrastructure rollbacks range from "re-apply old code in 5 minutes" to "restore RDS from backup and lose 30 minutes of data." Every destructive infrastructure change needs a documented rollback plan decided before the apply runs, not after it fails.

6. The two pipelines are independent by design. Keeping application and infrastructure deployments in separate pipelines is not just organizational — it is technical. The application pipeline needs to run 10 times a day without waiting for infrastructure approval. The infrastructure pipeline needs careful human review that would block application deploys if they were coupled. Independence is the right design.

What happens when they collide? If the infra pipeline is mid-apply when the app pipeline tries to deploy, the app deploy still works — it pushes a new image and updates the image_tag variable in HCP Terraform, which queues a run. That run waits for the in-flight infra apply to finish (HCP Terraform serializes runs per workspace) and then executes. The two pipelines coordinate through workspace queueing, not direct dependency.

7. HCP Terraform solves coordination, not configuration. The S3 + DynamoDB + GitHub Actions setup covers the technical requirements. What HCP Terraform adds is coordination: who approved this run, what was the full plan, which workspace is this configuration deployed to, what policies must pass before apply. For a solo engineer, these questions are trivial. For a team of 10 across 3 AWS accounts, they are the difference between an auditable system and organized chaos.

If you only do one thing from this post: pin every module to a semantic version tag and turn on speculative plans on PRs. Version pinning eliminates the entire "the module changed under us" failure class. Speculative plans put the infrastructure diff in front of the reviewer at the moment of decision, instead of after the merge. Together they catch the majority of production incidents before they ever reach an apply.

Key Terms

We have gone a long way

Twenty-two days in, the FastAPI stack is a production-grade system: containerized or EC2-based compute behind an ALB, RDS with encrypted storage and prevent_destroy, secrets in Secrets Manager, multi-region with Route 53 failover, tested with a full pyramid from static analysis to E2E, automatically deployed via CI/CD, with Sentinel enforcing security policies before every production apply.

The book is finished. The remaining eight days of the challenge move into hands-on territory — building the pieces that are still outstanding (the private networking module, advanced patterns), and applying everything to a real project rather than a controlled learning sequence.

The private networking module is next: VPC, public subnets, private subnets, database subnets, NAT Gateway, Internet Gateway, and route tables. It is the foundation that every other module in the stack should be using but currently is not, because the series started with the default VPC to reduce complexity. Fixing that gap is the natural first project.

This post is part of a 30-day Terraform learning journey.