Day 26: Building a Scalable Web Application with Terraform, EC2, and Auto Scaling

Overview

Architecture

                           Internet
                               │
                               ▼
                  ┌────────────────────────┐
                  │  Application Load      │   Public subnets, multi-AZ
                  │  Balancer (ALB)        │   HTTP → HTTPS redirect
                  │  Health check: /health │
                  └───────────┬────────────┘
                              │
            ┌─────────────────┼─────────────────┐
            │                 │                 │
        ┌───▼───┐         ┌───▼───┐         ┌───▼───┐
        │ EC2 1 │         │ EC2 2 │         │ EC2 N │   Auto Scaling Group
        │ (web) │         │ (web) │         │ (web) │   min=2, max=6, desired=2
        └───┬───┘         └───┬───┘         └───┬───┘   IMDSv2 enforced
            │                 │                 │
            └─────────────────┼─────────────────┘
                              │
                              ▼
                  ┌────────────────────────┐
                  │  CloudWatch Alarms     │   ASG average CPU
                  │  CPU > 70% (2m)        │   → scale-out policy (+1)
                  │  CPU < 30% (5m)        │   → scale-in  policy (−1)
                  └────────────────────────┘

Three things to note before writing any HCL:

1. The ALB lives in public subnets; the EC2 instances live in private subnets. Only the ALB has a public IP. Instances get traffic exclusively through the ALB target group. This is the standard production pattern — no instance is directly reachable from the internet.
2. The Auto Scaling Group is the source of truth for instance count, not the launch template. The launch template is a recipe (AMI, instance type, user data, IAM profile). The ASG is the runner that creates and destroys instances from that recipe based on the desired capacity and scaling alarms.
3. Scaling decisions are driven by CloudWatch alarms, not Terraform. Terraform creates the alarm and the scaling policy. From then on, AWS evaluates the metric and adjusts capacity — Terraform is not in the loop to understand as terraform is just an IAC which helps you only to provision these resources. This is critical: the ASG desired_capacity will drift from whatever you wrote in .tf as soon as a scaling event fires. The fix (covered in Gotchas) is lifecycle { ignore_changes = [desired_capacity] }.

Project Structure

scalable-web-app/
├── modules/
│   ├── ec2/                  ← launch template + security group
│   │   ├── main.tf
│   │   ├── user_data.sh.tpl
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── versions.tf
│   ├── alb/                  ← ALB + listener + target group
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── versions.tf
│   └── asg/                  ← ASG + scaling policies + CloudWatch alarms
│       ├── main.tf
│       ├── variables.tf
│       ├── outputs.tf
│       └── versions.tf
└── envs/
    └── dev/
        ├── backend.tf        ← S3 + DynamoDB remote state
        ├── providers.tf
        ├── main.tf           ← calls all three modules
        ├── variables.tf
        ├── outputs.tf
        └── terraform.tfvars

The split into modules/ (reusable) and envs/dev/ (environment-specific) is the same layout used since Day 9 and refined through Day 21. New environments (envs/staging, envs/prod) reuse the same modules with different variables.

The ec2 Module

The ec2 module produces a launch template, not actual instances. The ASG owns the instances; the launch template tells the ASG how to build each one.

modules/ec2/main.tf

data "aws_ami" "amazon_linux_2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }
}

# Security group: allow HTTP from the ALB security group only (no public ingress).
resource "aws_security_group" "web" {
  name_prefix = "${var.name_prefix}-web-"
  description = "Web tier — ingress from ALB only"
  vpc_id      = var.vpc_id

  ingress {
    description     = "HTTP from ALB"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [var.alb_security_group_id]
  }

  egress {
    description = "All egress"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = var.tags

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_launch_template" "web" {
  name_prefix   = "${var.name_prefix}-web-"
  image_id      = data.aws_ami.amazon_linux_2023.id
  instance_type = var.instance_type

  vpc_security_group_ids = [aws_security_group.web.id]

  iam_instance_profile {
    name = var.instance_profile_name
  }

  # IMDSv2 enforced — no SSRF token theft via IMDSv1.
  metadata_options {
    http_tokens                 = "required"
    http_endpoint               = "enabled"
    http_put_response_hop_limit = 1
  }

  user_data = base64encode(templatefile("${path.module}/user_data.sh.tpl", {
    app_name = var.name_prefix
  }))

  tag_specifications {
    resource_type = "instance"
    tags          = merge(var.tags, { Name = "${var.name_prefix}-web" })
  }

  lifecycle {
    create_before_destroy = true
  }
}

modules/ec2/user_data.sh.tpl

#!/bin/bash
set -euxo pipefail

dnf update -y
dnf install -y nginx

# Minimal landing page that includes the instance ID so we can prove the
# load balancer is round-robining traffic across multiple instances.
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $(curl -s -X PUT 'http://169.254.169.254/latest/api/token' -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')" http://169.254.169.254/latest/meta-data/instance-id)

cat > /usr/share/nginx/html/index.html <<EOF
<!doctype html>
<html><body>
  <h1>${app_name}</h1>
  <p>Served by instance: $INSTANCE_ID</p>
</body></html>
EOF

# Health endpoint used by the ALB target group.
echo 'ok' > /usr/share/nginx/html/health

systemctl enable nginx
systemctl start nginx

The IMDSv2 token dance is required because metadata_options.http_tokens = "required" was set on the launch template — IMDSv1 calls (a bare curl http://169.254.169.254/...) would fail with 401.

modules/ec2/variables.tf and outputs.tf

# variables.tf
variable "name_prefix"              { type = string }
variable "vpc_id"                   { type = string }
variable "alb_security_group_id"    { type = string }
variable "instance_type"            { type = string; default = "t3.micro" }
variable "instance_profile_name"    { type = string; default = null }
variable "tags"                     { type = map(string); default = {} }

# outputs.tf
output "launch_template_id"      { value = aws_launch_template.web.id }
output "launch_template_version" { value = aws_launch_template.web.latest_version }
output "security_group_id"       { value = aws_security_group.web.id }

The alb Module

modules/alb/main.tf

# Security group: allow HTTP/HTTPS from anywhere.
resource "aws_security_group" "alb" {
  name_prefix = "${var.name_prefix}-alb-"
  description = "ALB — public ingress on 80/443"
  vpc_id      = var.vpc_id

  ingress {
    description = "HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTPS"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "All egress (to web tier)"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = var.tags

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb" "this" {
  name               = "${var.name_prefix}-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = var.public_subnet_ids

  enable_deletion_protection = var.environment == "prod" ? true : false
  drop_invalid_header_fields = true   # drop malformed headers — OWASP recommendation

  tags = var.tags
}

resource "aws_lb_target_group" "web" {
  name        = "${var.name_prefix}-web-tg"
  port        = 80
  protocol    = "HTTP"
  target_type = "instance"
  vpc_id      = var.vpc_id

  health_check {
    enabled             = true
    path                = "/health"
    protocol            = "HTTP"
    matcher             = "200"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 2   # ALB default — explicit for clarity
  }

  deregistration_delay = 30   # default is 300s; 30s is more responsive for stateless apps

  tags = var.tags

  lifecycle {
    create_before_destroy = true
  }
}

# HTTP listener — redirects to HTTPS in prod, serves directly in dev.
resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.this.arn
  port              = 80
  protocol          = "HTTP"

  dynamic "default_action" {
    for_each = var.environment == "prod" ? [1] : []
    content {
      type = "redirect"
      redirect {
        port        = "443"
        protocol    = "HTTPS"
        status_code = "HTTP_301"
      }
    }
  }

  dynamic "default_action" {
    for_each = var.environment == "prod" ? [] : [1]
    content {
      type             = "forward"
      target_group_arn = aws_lb_target_group.web.arn
    }
  }
}

modules/alb/outputs.tf

output "alb_arn"               { value = aws_lb.this.arn }
output "alb_dns_name"          { value = aws_lb.this.dns_name }
output "alb_zone_id"           { value = aws_lb.this.zone_id }
output "alb_security_group_id" { value = aws_security_group.alb.id }
output "target_group_arn"      { value = aws_lb_target_group.web.arn }

The asg Module

This is the module that closes the loop. It owns the Auto Scaling Group, attaches it to the launch template and target group, and creates the CloudWatch alarms that drive scaling.

modules/asg/main.tf

resource "aws_autoscaling_group" "web" {
  name                      = "${var.name_prefix}-asg"
  vpc_zone_identifier       = var.private_subnet_ids
  target_group_arns         = [var.target_group_arn]
  health_check_type         = "ELB"   # ASG uses ALB health checks, not just EC2 status
  health_check_grace_period = 60

  min_size         = var.min_size
  max_size         = var.max_size
  desired_capacity = var.desired_capacity

  launch_template {
    id      = var.launch_template_id
    version = var.launch_template_version
  }

  # Rolling instance refresh: when the launch template changes (new AMI, new
  # user data), the ASG replaces instances 50% at a time instead of all at once.
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 50
      instance_warmup        = 60
    }
  }

  tag {
    key                 = "Name"
    value               = "${var.name_prefix}-web"
    propagate_at_launch = true
  }

  dynamic "tag" {
    for_each = var.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }

  # CRITICAL: ignore desired_capacity drift from scaling actions.
  # Without this, every Terraform plan after a scale event would try to reset
  # the ASG to the original desired_capacity, undoing the scale.
  lifecycle {
    ignore_changes        = [desired_capacity]
    create_before_destroy = true
  }
}

# ── Scaling policies ────────────────────────────────────────────────────────
# Two simple-step policies: scale out (+1) when CPU is high, scale in (-1) when low.
# For most production workloads, prefer aws_autoscaling_policy with policy_type =
# "TargetTrackingScaling" and a target value (e.g., maintain 50% CPU). Step scaling
# is shown here because it makes the alarm → policy mapping explicit.

resource "aws_autoscaling_policy" "scale_out" {
  name                   = "${var.name_prefix}-scale-out"
  scaling_adjustment     = 1
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 120
  autoscaling_group_name = aws_autoscaling_group.web.name
}

resource "aws_autoscaling_policy" "scale_in" {
  name                   = "${var.name_prefix}-scale-in"
  scaling_adjustment     = -1
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 300
  autoscaling_group_name = aws_autoscaling_group.web.name
}

# ── CloudWatch alarms ───────────────────────────────────────────────────────
# Alarms watch ASG-aggregated CPU. Period = 60s, two consecutive breaches required
# for scale-out (faster to react) and five for scale-in (slower to react).

resource "aws_cloudwatch_metric_alarm" "cpu_high" {
  alarm_name          = "${var.name_prefix}-cpu-high"
  alarm_description   = "ASG average CPU > 70% for 2 minutes → scale out"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  statistic           = "Average"
  period              = 60
  evaluation_periods  = 2
  threshold           = 70
  comparison_operator = "GreaterThanThreshold"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.web.name
  }

  alarm_actions = [aws_autoscaling_policy.scale_out.arn]
}

resource "aws_cloudwatch_metric_alarm" "cpu_low" {
  alarm_name          = "${var.name_prefix}-cpu-low"
  alarm_description   = "ASG average CPU < 30% for 5 minutes → scale in"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  statistic           = "Average"
  period              = 60
  evaluation_periods  = 5
  threshold           = 30
  comparison_operator = "LessThanThreshold"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.web.name
  }

  alarm_actions = [aws_autoscaling_policy.scale_in.arn]
}

modules/asg/outputs.tf

output "asg_name"            { value = aws_autoscaling_group.web.name }
output "asg_arn"             { value = aws_autoscaling_group.web.arn }
output "scale_out_alarm_arn" { value = aws_cloudwatch_metric_alarm.cpu_high.arn }
output "scale_in_alarm_arn"  { value = aws_cloudwatch_metric_alarm.cpu_low.arn }

Root Composition

The envs/dev/ directory wires the three modules together, plus the remote state backend.

envs/dev/backend.tf

terraform {
  required_version = ">= 1.6.0"

  backend "s3" {
    bucket         = "mnourdine-tf-state"
    key            = "dev/scalable-web-app/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

envs/dev/providers.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.region
  default_tags {
    tags = {
      Project     = "scalable-web-app"
      Environment = "dev"
      ManagedBy   = "terraform"
    }
  }
}

envs/dev/main.tf

# Use existing VPC and subnets. In a greenfield deployment, this is where
# the networking module from Day 27 will be wired in. For Day 26, we read
# the default VPC's public subnets and reuse them as both ALB and "private"
# subnets — a deliberate simplification, see Gotchas.
data "aws_vpc" "default" {
  default = true
}

data "aws_subnets" "default" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }
}

module "alb" {
  source = "../../modules/alb"

  name_prefix       = "${var.project_name}-${var.environment}"
  vpc_id            = data.aws_vpc.default.id
  public_subnet_ids = data.aws_subnets.default.ids
  environment       = var.environment
}

module "ec2" {
  source = "../../modules/ec2"

  name_prefix           = "${var.project_name}-${var.environment}"
  vpc_id                = data.aws_vpc.default.id
  alb_security_group_id = module.alb.alb_security_group_id
  instance_type         = var.instance_type
}

module "asg" {
  source = "../../modules/asg"

  name_prefix             = "${var.project_name}-${var.environment}"
  private_subnet_ids      = data.aws_subnets.default.ids
  target_group_arn        = module.alb.target_group_arn
  launch_template_id      = module.ec2.launch_template_id
  launch_template_version = module.ec2.launch_template_version
  min_size                = 2
  max_size                = 6
  desired_capacity        = 2
}

envs/dev/outputs.tf

output "alb_dns_name" {
  description = "Public DNS name of the ALB — the entry point to the app"
  value       = module.alb.alb_dns_name
}

output "asg_name" {
  description = "Auto Scaling Group name — used for inspecting scaling activity"
  value       = module.asg.asg_name
}

Verifying It Scales

Initial deploy

cd envs/dev
terraform init
terraform plan
terraform apply

# Wait ~2 minutes for instances to launch and pass health checks
ALB=$(terraform output -raw alb_dns_name)
ASG=$(terraform output -raw asg_name)

# Should return an HTML page with the instance ID. Hit it a few times —
# the instance ID should rotate as the ALB round-robins across targets.
curl http://$ALB
curl http://$ALB
curl http://$ALB

Generate load to trigger scale-out

The fastest way to drive CPU up is to SSH (or use SSM Session Manager — see Day 18) into one of the instances and run a CPU burner:

# Log into one instance via SSM Session Manager:
aws ssm start-session --target <instance-id>

# Then on the instance:
sudo dnf install -y stress-ng
stress-ng --cpu 0 --timeout 600   # peg all CPUs for 10 minutes

Or generate load against the ALB from your laptop, which spreads CPU across all instances:

# Apache Bench — 100 concurrent connections, 100k requests
ab -n 100000 -c 100 http://$ALB/

Watch the ASG react

# Live view of ASG capacity
watch -n 5 "aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names $ASG \
  --query 'AutoScalingGroups[0].[DesiredCapacity,length(Instances)]' \
  --output text"

# Recent scaling activities (most recent first)
aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name $ASG \
  --max-items 5 \
  --query 'Activities[*].[StartTime,Description,StatusCode]' \
  --output table

# Alarm state
aws cloudwatch describe-alarms \
  --alarm-names "${PROJECT}-${ENV}-cpu-high" "${PROJECT}-${ENV}-cpu-low" \
  --query 'MetricAlarms[*].[AlarmName,StateValue,StateReason]' \
  --output table

What you should see, in order:

1. CPU climbs above 70%; the cpu-high alarm transitions from OK to ALARM after two evaluation periods (~2 minutes).
2. The scale-out policy fires; ASG DesiredCapacity increases by 1.
3. AWS launches a new instance from the launch template; it takes ~45–60 seconds to boot, install nginx, and pass the health check.
4. The new instance is added to the ALB target group; load disperses; per-instance CPU drops.
5. If load remains high, the alarm fires again after the cooldown (120s), adding another instance, up to max_size = 6.
6. Stop the load generator. After ~5 minutes of CPU < 30%, the scale-in policy fires and removes one instance at a time, with a 5-minute cooldown between each, down to min_size = 2.

Best Practices Recap

This deploy uses every infrastructure convention built across the series:

Gotchas

The ASG desired_capacity will drift from your .tf file. This is by design — scaling actions change capacity, and Terraform shouldn't undo them. The lifecycle { ignore_changes = [desired_capacity] } in the asg module handles this. If you forget it, every terraform plan after a scale event will show a "change" that resets capacity to your hardcoded number, and terraform apply will undo the autoscaler. This is the single most common ASG mistake.

Default VPC subnets are all public. The envs/dev/main.tf uses the default VPC's subnets for both the ALB and the ASG. That works for a learning deploy but it means instances have public IPs and aren't actually behind a private network. Day 27 introduces a real networking module with separate public and private subnets; until then, treat this as a known shortcut. The security group still restricts ingress to the ALB SG, so it isn't insecure — just architecturally simplified.

Health checks: health_check_type = "ELB" is critical. The ASG default is "EC2", which only checks if the instance is alive at the hypervisor level. With "ELB", the ASG also respects ALB target group health checks — if nginx crashes but the instance is still running, the ASG will replace it. Without this, broken instances hang around as "healthy" until something else terminates them.

Step scaling vs target tracking. This module uses step scaling (alarm fires → add or remove fixed number of instances) because it makes the alarm-policy wiring explicit and easy to reason about. For most production workloads, target tracking (policy_type = "TargetTrackingScaling" with a target value like "maintain 50% CPU average") is simpler and self-tuning — AWS handles the math. Use step scaling when you need precise control over the scaling steps; use target tracking when you just want capacity to track demand.

Cooldowns matter. Scale-out cooldown is 120s, scale-in cooldown is 300s. Scaling in slowly is deliberate: a sudden traffic dip might be a 30-second cache warmup, not a real demand drop, and removing an instance only to immediately add it back is wasteful and disrupts in-flight requests. Scaling out fast and scaling in slow is the standard asymmetric pattern.

terraform destroy order. Because the ALB target group references the ASG and the ASG references the launch template, terraform destroy walks the dependency graph in reverse: alarms → policies → ASG (which terminates instances, ~1 minute) → target group → ALB → launch template → security groups. Total destroy time is typically 3–5 minutes. If a destroy hangs, the most likely cause is the ASG waiting for instances to be marked terminated by the EC2 API.

Cost expectations. A small dev deploy of this stack runs around $20–30/month: ~$15 for the ALB ($0.0225/hour fixed plus $0.008 per LCU), ~$8 for two t3.micro instances at on-demand pricing, near-zero for CloudWatch alarms (10 alarms/month free per account). The two costs that scale with traffic are ALB LCUs (new connections, active connections, processed bytes) and EC2 instance-hours during scale-out events.

What have we learned here?

The scalable web application is end-to-end production-shaped: a public ALB terminating HTTP/HTTPS, an Auto Scaling Group of private EC2 instances behind it, and a CloudWatch-driven feedback loop that adds or removes capacity as traffic shifts. Three small modules — ec2, alb, asg — compose into one environment, all behind a remote state backend with locking.

Two pieces are still simplified relative to a real production stack: the deployment uses the default VPC's subnets (no real public/private separation), and the EC2 instances run a static nginx page rather than a real application image. Day 27 fixes the first by introducing a proper networking module — VPC, public and private subnets, NAT Gateway, route tables — which is the foundation that every module from Day 9 onward should have been built on top of. The second gap (real app deployment) was already covered conceptually in Day 22: the Terraform pipeline manages the ASG, while a separate application pipeline updates the launch template's user data or AMI, triggering an instance refresh.

The pattern — small modules, remote state, IMDSv2, security-group references between tiers, scaling owned by AWS rather than Terraform — is the same pattern that has run through every production deploy in the series. The novelty here is the closed-loop scaling: from this point on, the system reacts to load without a human in the loop.

This post is part of a 30-day Terraform learning journey.