Day 27: Building a Multi-Region, Fault-Tolerant 3-Tier Infrastructure with AWS and Terraform

Overview

Architecture

                                    ┌────────────────────┐
                                    │   Route 53 zone    │
                                    │  app.example.com   │
                                    │  failover record   │
                                    └─────────┬──────────┘
                                              │
                            ┌─────────────────┴─────────────────┐
                       PRIMARY                              SECONDARY
                  (health check OK)                  (used only on failover)
                            │                                   │
                            ▼                                   ▼
              ┌──────────────────────────┐        ┌──────────────────────────┐
              │     us-east-1 region     │        │     us-west-2 region     │
              │                          │        │                          │
              │  ┌────────────────────┐  │        │  ┌────────────────────┐  │
              │  │       ALB          │  │        │  │       ALB          │  │
              │  └─────────┬──────────┘  │        │  └─────────┬──────────┘  │
              │            │             │        │            │             │
              │  ┌─────────▼──────────┐  │        │  ┌─────────▼──────────┐  │
              │  │   ASG: web tier    │  │        │  │   ASG: web tier    │  │
              │  │   (private subnets)│  │        │  │   (private subnets)│  │
              │  └─────────┬──────────┘  │        │  └─────────┬──────────┘  │
              │            │             │        │            │             │
              │  ┌─────────▼──────────┐  │        │  ┌─────────▼──────────┐  │
              │  │  RDS PRIMARY       │  │ async  │  │  RDS READ REPLICA  │  │
              │  │  Multi-AZ          │──┼────────┼─▶│  (promotable)      │  │
              │  └────────────────────┘  │repl.   │  └────────────────────┘  │
              │                          │        │                          │
              │  ┌────────────────────┐  │ S3 CRR │  ┌────────────────────┐  │
              │  │  S3 assets bucket  │──┼────────┼─▶│  S3 replica bucket │  │
              │  └────────────────────┘  │        │  └────────────────────┘  │
              └──────────────────────────┘        └──────────────────────────┘

Three things to keep in mind before any HCL:

1. Active-passive, not active-active. Both regions are deployed and warm, but only us-east-1 serves traffic under normal conditions. The Route 53 failover policy sends 100% of requests to the primary while its health check is HEALTHY, and switches to the secondary the moment the health check fails. Active-active (latency-based or weighted routing across both regions simultaneously) is a different problem because it requires bidirectional database replication or a globally-consistent data store — out of scope for this blog post.

2. The database is the hard part. RDS cross-region read replicas are asynchronous. The replica lags the primary by some amount of time (typically seconds, occasionally minutes under load). On failover, you promote the replica to a standalone primary — and any writes that hadn't replicated yet are lost. Plan for this; it's not a bug, it's the cost of cross-region durability over synchronous latency.

3. Failover is a one-way door (sort of). Once you promote the west replica to primary, the original east primary is no longer related to it. Failing back means rebuilding replication in the opposite direction (or restoring from snapshot). This is why most teams treat regional failover as a planned-for but rarely-exercised event.

Project Structure

multi-region-ha/
├── modules/
│   ├── vpc/                 ← public/private subnets, NAT, route tables
│   ├── web/                 ← ALB + ASG (composes Day 26's alb/ec2/asg modules)
│   ├── rds/                 ← primary OR replica, depending on inputs
│   ├── dns/                 ← Route 53 zone + health checks + failover records
│   └── assets/              ← S3 bucket + replication configuration
└── envs/
    └── prod/
        ├── backend.tf       ← S3 + DynamoDB remote state
        ├── providers.tf     ← two aliased providers: aws.east, aws.west
        ├── main.tf          ← invokes each module twice (once per region)
        ├── variables.tf
        ├── outputs.tf
        └── terraform.tfvars

The big shift from Day 26 is providers.tf: instead of one default aws provider, the root config declares two aliased providers (aws.east and aws.west), and each module call passes the appropriate one in. Provider aliases for multi-region were introduced back on Day 14 — same mechanism, larger blast radius.

The vpc Module

A real production VPC, not the default-VPC shortcut from Day 26. Two AZs, public subnets for the ALB, private subnets for EC2 and RDS, one NAT Gateway per AZ for outbound traffic from private instances.

modules/vpc/main.tf

data "aws_availability_zones" "this" {
  state = "available"
}

locals {
  azs = slice(data.aws_availability_zones.this.names, 0, 2)
}

resource "aws_vpc" "this" {
  cidr_block           = var.cidr_block
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = merge(var.tags, { Name = "${var.name_prefix}-vpc" })
}

resource "aws_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id
  tags   = merge(var.tags, { Name = "${var.name_prefix}-igw" })
}

# Public subnets — one per AZ
resource "aws_subnet" "public" {
  for_each                = { for idx, az in local.azs : az => idx }
  vpc_id                  = aws_vpc.this.id
  cidr_block              = cidrsubnet(var.cidr_block, 8, each.value)
  availability_zone       = each.key
  map_public_ip_on_launch = true

  tags = merge(var.tags, {
    Name = "${var.name_prefix}-public-${each.key}"
    Tier = "public"
  })
}

# Private subnets — one per AZ, offset by 100 to avoid CIDR collision
resource "aws_subnet" "private" {
  for_each          = { for idx, az in local.azs : az => idx + 100 }
  vpc_id            = aws_vpc.this.id
  cidr_block        = cidrsubnet(var.cidr_block, 8, each.value)
  availability_zone = each.key

  tags = merge(var.tags, {
    Name = "${var.name_prefix}-private-${each.key}"
    Tier = "private"
  })
}

# One NAT Gateway per AZ — survives single-AZ failure, costs ~$32/month each.
# For dev environments, drop to one NAT in a single AZ to save ~$32/month.
resource "aws_eip" "nat" {
  for_each = aws_subnet.public
  domain   = "vpc"
  tags     = merge(var.tags, { Name = "${var.name_prefix}-nat-${each.key}" })
}

resource "aws_nat_gateway" "this" {
  for_each      = aws_subnet.public
  allocation_id = aws_eip.nat[each.key].id
  subnet_id     = each.value.id

  tags = merge(var.tags, { Name = "${var.name_prefix}-nat-${each.key}" })

  depends_on = [aws_internet_gateway.this]
}

# Public route table — one, shared across both public subnets
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.this.id
  tags   = merge(var.tags, { Name = "${var.name_prefix}-public-rt" })
}

resource "aws_route" "public_internet" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.this.id
}

resource "aws_route_table_association" "public" {
  for_each       = aws_subnet.public
  subnet_id      = each.value.id
  route_table_id = aws_route_table.public.id
}

# Private route tables — one per AZ, each pointing at its own NAT Gateway.
# Sharing one private route table across AZs is a common mistake: it routes
# all private traffic through one NAT, defeating the per-AZ redundancy.
resource "aws_route_table" "private" {
  for_each = aws_subnet.private
  vpc_id   = aws_vpc.this.id
  tags     = merge(var.tags, { Name = "${var.name_prefix}-private-rt-${each.key}" })
}

resource "aws_route" "private_nat" {
  for_each               = aws_route_table.private
  route_table_id         = each.value.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.this[each.key].id
}

resource "aws_route_table_association" "private" {
  for_each       = aws_subnet.private
  subnet_id      = each.value.id
  route_table_id = aws_route_table.private[each.key].id
}

modules/vpc/outputs.tf

output "vpc_id"             { value = aws_vpc.this.id }
output "public_subnet_ids"  { value = [for s in aws_subnet.public  : s.id] }
output "private_subnet_ids" { value = [for s in aws_subnet.private : s.id] }

The web Module

The web tier is essentially the Day 26 stack repackaged: an ALB in public subnets, an ASG of EC2 instances in private subnets, and CloudWatch alarms driving scaling. The Day 27 web module just composes the three Day 26 modules (alb, ec2, asg) and adds a database connection string injected via user data.

modules/web/main.tf

module "alb" {
  source = "../../../scalable-web-app/modules/alb"

  name_prefix       = var.name_prefix
  vpc_id            = var.vpc_id
  public_subnet_ids = var.public_subnet_ids
  environment       = "prod"
}

module "ec2" {
  source = "../../../scalable-web-app/modules/ec2"

  name_prefix           = var.name_prefix
  vpc_id                = var.vpc_id
  alb_security_group_id = module.alb.alb_security_group_id
  instance_type         = var.instance_type

  # Database endpoint injected into user_data so the app knows where to connect.
  # In a real deploy this would come from SSM Parameter Store or Secrets Manager,
  # not be baked into the launch template — see Gotchas.
  extra_user_data_vars = {
    db_endpoint = var.db_endpoint
    db_name     = var.db_name
  }
}

module "asg" {
  source = "../../../scalable-web-app/modules/asg"

  name_prefix             = var.name_prefix
  private_subnet_ids      = var.private_subnet_ids
  target_group_arn        = module.alb.target_group_arn
  launch_template_id      = module.ec2.launch_template_id
  launch_template_version = module.ec2.launch_template_version
  min_size                = var.min_size
  max_size                = var.max_size
  desired_capacity        = var.desired_capacity
}

modules/web/outputs.tf

output "alb_dns_name" { value = module.alb.alb_dns_name }
output "alb_zone_id"  { value = module.alb.alb_zone_id }
output "alb_arn"      { value = module.alb.alb_arn }

The rds Module

The RDS module has two modes, controlled by a single is_replica flag:

• Primary mode: creates an aws_db_instance with backups enabled (required for replication), Multi-AZ for in-region HA, and a security group that allows ingress from the web tier.
• Replica mode: creates an aws_db_instance with replicate_source_db set to the ARN of the primary (an ARN is required for cross-region; only intra-region uses the bare identifier).

modules/rds/main.tf

resource "aws_db_subnet_group" "this" {
  name       = "${var.name_prefix}-db-subnets"
  subnet_ids = var.private_subnet_ids
  tags       = var.tags
}

resource "aws_security_group" "db" {
  name_prefix = "${var.name_prefix}-db-"
  description = "RDS — ingress from web tier on 5432"
  vpc_id      = var.vpc_id

  ingress {
    description     = "Postgres from web tier"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [var.web_security_group_id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = var.tags
}

# ── PRIMARY ──────────────────────────────────────────────────────────────────
resource "aws_db_instance" "primary" {
  count = var.is_replica ? 0 : 1

  identifier              = "${var.name_prefix}-primary"
  engine                  = "postgres"
  engine_version          = var.engine_version
  instance_class          = var.instance_class
  allocated_storage       = var.allocated_storage
  storage_encrypted       = true
  kms_key_id              = var.kms_key_arn

  db_name                 = var.db_name
  username                = var.master_username
  manage_master_user_password = true   # rotates password via Secrets Manager

  db_subnet_group_name    = aws_db_subnet_group.this.name
  vpc_security_group_ids  = [aws_security_group.db.id]
  multi_az                = true       # in-region HA via synchronous standby

  # Backups MUST be enabled (retention >= 1 day) for replication to work.
  backup_retention_period = 7
  backup_window           = "03:00-04:00"

  deletion_protection     = true
  skip_final_snapshot     = false
  final_snapshot_identifier = "${var.name_prefix}-final-${formatdate("YYYYMMDDhhmmss", timestamp())}"

  performance_insights_enabled = true
  enabled_cloudwatch_logs_exports = ["postgresql"]

  tags = var.tags

  lifecycle {
    ignore_changes = [final_snapshot_identifier]
  }
}

# ── CROSS-REGION READ REPLICA ────────────────────────────────────────────────
resource "aws_db_instance" "replica" {
  count = var.is_replica ? 1 : 0

  identifier              = "${var.name_prefix}-replica"
  instance_class          = var.instance_class
  storage_encrypted       = true
  kms_key_id              = var.kms_key_arn   # KMS key in the REPLICA region

  # Cross-region replication requires the source ARN, not just the identifier.
  replicate_source_db     = var.source_db_arn

  db_subnet_group_name    = aws_db_subnet_group.this.name
  vpc_security_group_ids  = [aws_security_group.db.id]

  # Replicas don't take backups in the traditional sense, but enabling this
  # turns the replica into a candidate for further chained replication if
  # you ever need a third region.
  backup_retention_period = 7

  # Read replicas cannot be Multi-AZ until they're promoted. Set this to
  # true to take effect ONLY after promotion.
  multi_az                = false

  skip_final_snapshot     = true   # replicas have no final snapshot semantics
  deletion_protection     = false  # easier to tear down standby in tests

  performance_insights_enabled = true

  tags = var.tags
}

modules/rds/outputs.tf

output "endpoint" {
  value = var.is_replica ? aws_db_instance.replica[0].endpoint : aws_db_instance.primary[0].endpoint
}

output "arn" {
  value = var.is_replica ? aws_db_instance.replica[0].arn : aws_db_instance.primary[0].arn
}

output "db_security_group_id" {
  value = aws_security_group.db.id
}

The dns Module

Route 53 ties the two regions together. Two failover-routing-policy A-records share the same name (app.example.com) — one PRIMARY, one SECONDARY — each pointing at its region's ALB via an alias. Each record references a Route 53 health check. Route 53 evaluates the primary health check; if it fails, traffic flips to the secondary record.

modules/dns/main.tf

data "aws_route53_zone" "this" {
  name = var.zone_name
}

# ── Health checks ────────────────────────────────────────────────────────────
# One per region. Route 53 health checkers pull the URL from ~16 global
# locations. Failure = "more than 18% of checkers report unhealthy".
resource "aws_route53_health_check" "primary" {
  fqdn              = var.primary_alb_dns
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30

  tags = merge(var.tags, { Name = "${var.name_prefix}-primary-hc" })
}

resource "aws_route53_health_check" "secondary" {
  fqdn              = var.secondary_alb_dns
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30

  tags = merge(var.tags, { Name = "${var.name_prefix}-secondary-hc" })
}

# ── Failover records ─────────────────────────────────────────────────────────
resource "aws_route53_record" "primary" {
  zone_id = data.aws_route53_zone.this.zone_id
  name    = var.record_name
  type    = "A"

  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id

  failover_routing_policy {
    type = "PRIMARY"
  }

  alias {
    name                   = var.primary_alb_dns
    zone_id                = var.primary_alb_zone_id
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "secondary" {
  zone_id = data.aws_route53_zone.this.zone_id
  name    = var.record_name
  type    = "A"

  set_identifier  = "secondary"
  health_check_id = aws_route53_health_check.secondary.id

  failover_routing_policy {
    type = "SECONDARY"
  }

  alias {
    name                   = var.secondary_alb_dns
    zone_id                = var.secondary_alb_zone_id
    evaluate_target_health = true
  }
}

The evaluate_target_health = true on each alias is doubly belt-and-braces: even if the Route 53 health check disagrees, the ALB's own target health is also factored in. If the ALB has zero healthy targets, the alias is treated as unhealthy.

The assets Module

S3 cross-region replication for static assets (images, JavaScript bundles, CSS). Source bucket in us-east-1, replica in us-west-2, both versioned (versioning is mandatory for replication).

modules/assets/main.tf

# ── Source bucket (us-east-1) ────────────────────────────────────────────────
resource "aws_s3_bucket" "source" {
  bucket = "${var.name_prefix}-assets-source"
  tags   = var.tags
}

resource "aws_s3_bucket_versioning" "source" {
  bucket = aws_s3_bucket.source.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "source" {
  bucket = aws_s3_bucket.source.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "source" {
  bucket                  = aws_s3_bucket.source.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# ── Destination bucket (us-west-2) — created by passing aws.west into module ─
resource "aws_s3_bucket" "destination" {
  provider = aws.replica
  bucket   = "${var.name_prefix}-assets-replica"
  tags     = var.tags
}

resource "aws_s3_bucket_versioning" "destination" {
  provider = aws.replica
  bucket   = aws_s3_bucket.destination.id
  versioning_configuration {
    status = "Enabled"
  }
}

# ── Replication role + policy ────────────────────────────────────────────────
resource "aws_iam_role" "replication" {
  name = "${var.name_prefix}-s3-replication"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "s3.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_policy" "replication" {
  name = "${var.name_prefix}-s3-replication"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetReplicationConfiguration",
          "s3:ListBucket",
        ]
        Resource = [aws_s3_bucket.source.arn]
      },
      {
        Effect = "Allow"
        Action = [
          "s3:GetObjectVersionForReplication",
          "s3:GetObjectVersionAcl",
          "s3:GetObjectVersionTagging",
        ]
        Resource = ["${aws_s3_bucket.source.arn}/*"]
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ReplicateObject",
          "s3:ReplicateDelete",
          "s3:ReplicateTags",
        ]
        Resource = ["${aws_s3_bucket.destination.arn}/*"]
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "replication" {
  role       = aws_iam_role.replication.name
  policy_arn = aws_iam_policy.replication.arn
}

# ── Replication configuration ────────────────────────────────────────────────
resource "aws_s3_bucket_replication_configuration" "this" {
  # Versioning must be enabled on BOTH buckets before this resource can be created.
  depends_on = [
    aws_s3_bucket_versioning.source,
    aws_s3_bucket_versioning.destination,
  ]

  role   = aws_iam_role.replication.arn
  bucket = aws_s3_bucket.source.id

  rule {
    id     = "replicate-everything"
    status = "Enabled"

    filter {}   # empty filter = match all objects

    delete_marker_replication {
      status = "Enabled"
    }

    destination {
      bucket        = aws_s3_bucket.destination.arn
      storage_class = "STANDARD"
    }
  }
}

The module declares a second provider configuration through the configuration_aliases mechanism — same pattern as Day 14's multi-region aws.us_east_1 example:

modules/assets/versions.tf

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source                = "hashicorp/aws"
      version               = "~> 5.0"
      configuration_aliases = [aws.replica]
    }
  }
}

Root Composition

The root config calls each module twice (once per region) with the appropriate provider passed in.

envs/prod/providers.tf

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  alias  = "east"
  region = "us-east-1"
  default_tags { tags = local.common_tags }
}

provider "aws" {
  alias  = "west"
  region = "us-west-2"
  default_tags { tags = local.common_tags }
}

envs/prod/main.tf

locals {
  name_prefix  = "${var.project_name}-prod"
  common_tags  = {
    Project     = var.project_name
    Environment = "prod"
    ManagedBy   = "terraform"
  }
}

# ── VPCs (one per region) ────────────────────────────────────────────────────
module "vpc_east" {
  source     = "../../modules/vpc"
  providers  = { aws = aws.east }
  name_prefix = "${local.name_prefix}-east"
  cidr_block  = "10.10.0.0/16"
}

module "vpc_west" {
  source     = "../../modules/vpc"
  providers  = { aws = aws.west }
  name_prefix = "${local.name_prefix}-west"
  cidr_block  = "10.20.0.0/16"   # MUST not overlap with east
}

# ── Database — primary in east, replica in west ──────────────────────────────
module "rds_primary" {
  source             = "../../modules/rds"
  providers          = { aws = aws.east }
  name_prefix        = "${local.name_prefix}-east"
  vpc_id             = module.vpc_east.vpc_id
  private_subnet_ids = module.vpc_east.private_subnet_ids
  web_security_group_id = module.web_east.web_security_group_id
  is_replica         = false
  db_name            = var.db_name
  master_username    = var.master_username
  kms_key_arn        = aws_kms_key.rds_east.arn
}

module "rds_replica" {
  source             = "../../modules/rds"
  providers          = { aws = aws.west }
  name_prefix        = "${local.name_prefix}-west"
  vpc_id             = module.vpc_west.vpc_id
  private_subnet_ids = module.vpc_west.private_subnet_ids
  web_security_group_id = module.web_west.web_security_group_id
  is_replica         = true
  source_db_arn      = module.rds_primary.arn
  kms_key_arn        = aws_kms_key.rds_west.arn   # KMS key in west region
}

# ── Web tier — one per region, each pointing at its local DB endpoint ────────
module "web_east" {
  source             = "../../modules/web"
  providers          = { aws = aws.east }
  name_prefix        = "${local.name_prefix}-east"
  vpc_id             = module.vpc_east.vpc_id
  public_subnet_ids  = module.vpc_east.public_subnet_ids
  private_subnet_ids = module.vpc_east.private_subnet_ids
  db_endpoint        = module.rds_primary.endpoint
  db_name            = var.db_name
  min_size           = 2
  max_size           = 6
  desired_capacity   = 2
}

module "web_west" {
  source             = "../../modules/web"
  providers          = { aws = aws.west }
  name_prefix        = "${local.name_prefix}-west"
  vpc_id             = module.vpc_west.vpc_id
  public_subnet_ids  = module.vpc_west.public_subnet_ids
  private_subnet_ids = module.vpc_west.private_subnet_ids
  db_endpoint        = module.rds_replica.endpoint   # read-only until promoted
  db_name            = var.db_name
  min_size           = 2
  max_size           = 6
  desired_capacity   = 2
}

# ── DNS failover ─────────────────────────────────────────────────────────────
# Route 53 hosted zones are global, so one provider is enough.
module "dns" {
  source                = "../../modules/dns"
  providers             = { aws = aws.east }
  name_prefix           = local.name_prefix
  zone_name             = var.zone_name
  record_name           = var.record_name
  primary_alb_dns       = module.web_east.alb_dns_name
  primary_alb_zone_id   = module.web_east.alb_zone_id
  secondary_alb_dns     = module.web_west.alb_dns_name
  secondary_alb_zone_id = module.web_west.alb_zone_id
}

# ── S3 cross-region replication ──────────────────────────────────────────────
module "assets" {
  source = "../../modules/assets"
  providers = {
    aws         = aws.east
    aws.replica = aws.west
  }
  name_prefix = local.name_prefix
}

envs/prod/outputs.tf

output "app_url" {
  description = "Failover-routed URL — DNS resolves to whichever region is healthy"
  value       = "http://${var.record_name}"
}

output "primary_alb_dns" { value = module.web_east.alb_dns_name }
output "secondary_alb_dns" { value = module.web_west.alb_dns_name }
output "primary_db_endpoint" { value = module.rds_primary.endpoint }
output "replica_db_endpoint" { value = module.rds_replica.endpoint }

Failover, In Practice

The whole design only earns its keep if failover actually works under stress. Two scenarios worth rehearsing:

Scenario 1: ALB or ASG failure in the primary region

This is the cheap failure mode. Simulate it by scaling the primary ASG to zero:

aws autoscaling update-auto-scaling-group \
  --region us-east-1 \
  --auto-scaling-group-name myapp-prod-east-asg \
  --min-size 0 --max-size 0 --desired-capacity 0

Within ~90 seconds (3 health checks × 30s) the Route 53 health check transitions to Unhealthy. Within another ~60 seconds — the default Route 53 record TTL — DNS resolvers worldwide stop returning the primary ALB and start returning the secondary ALB. End-to-end failover: roughly 2–3 minutes. Crucially, the application keeps working because the west region's web tier has been running the entire time with read-only access to the replica DB. If your app degrades gracefully under read-only mode (or if writes are infrequent), users barely notice.

To recover: scale the east ASG back up. Once health checks return to Healthy, traffic flips back automatically.

Scenario 2: Total loss of us-east-1

This is the expensive one. The primary ALB, ASG, and RDS primary are all unreachable. Two things have to happen:

1. Route 53 fails over DNS (automatic, same 2–3 minutes as above).

2. You promote the replica (manual, with data loss):

   aws rds promote-read-replica \
     --region us-west-2 \
     --db-instance-identifier myapp-prod-west-replica
   

   Promotion takes 5–10 minutes and breaks the replication relationship permanently. After promotion, the west DB is a standalone primary that accepts writes. You then reconfigure the west web tier's db_endpoint to point at it as a writable database (in this stack, that's already the case since the endpoint string doesn't change on promotion — only the read-only flag does).

The replica may lag the primary by anywhere from a few seconds to a few minutes depending on write volume and inter-region network conditions. The data written to the primary in that window is lost on promotion. If your application can't tolerate any data loss, you need a fundamentally different architecture (synchronous multi-region writes, e.g., Aurora Global Database with write forwarding, or a globally-distributed database like DynamoDB Global Tables) which we will discuss in another series :).

Things to Remember

KMS keys are regional. The CMK encrypting the primary RDS instance lives in us-east-1. The replica needs its own CMK in us-west-2 (var.kms_key_arn in the replica module call points at the west key). Cross-region replicas cannot use the source region's KMS key — RDS will refuse to start. Same constraint for S3 replication if you want SSE-KMS on the destination bucket.

S3 bucket names are globally unique, even across replication. ${name_prefix}-assets-source and ${name_prefix}-assets-replica need different names. Don't try to use the same bucket name in two regions; that's not a thing in S3. With the new update of AWS For Regional Unique buckets names you can bypass this constrant.

RDS replica promotion is one-way. Once promoted, you cannot demote the replica back to a replica of the original primary. To "fail back" to east, you have to either: (a) restore the east DB from a snapshot of the now-primary west DB and rebuild replication in the opposite direction, or (b) accept that west is the new primary and reconfigure replication to make east the new replica. Most teams plan for failover but rarely fail back; they let west become the new primary indefinitely.

Route 53 health checks bill per check, per region. Default health checks are $0.50/month each, plus $0.50/month per additional region the checker runs from. Two health checks in this stack ≈ $1–2/month. Cheap, but if you start checking dozens of endpoints with string_match enabled you'll see it on the bill.

evaluate_target_health on alias records can cause flapping. If the ALB's target group oscillates between healthy and unhealthy (typical during instance refresh), Route 53 may briefly remove the alias from DNS responses. Set the ALB target group's health check intervals and thresholds to avoid false negatives — unhealthy_threshold = 3 minimum.

The two regions' VPCs cannot share CIDRs. vpc_east uses 10.10.0.0/16; vpc_west uses 10.20.0.0/16. If you ever want to peer them (or join them via Transit Gateway), they MUST be non-overlapping. Pick non-overlapping CIDRs from day one even if you don't peer immediately.

NAT Gateways are the silent budget killer. Two NAT Gateways per region × two regions = four NAT Gateways × $32/month each = **$128/month** before any data transfer charges. For dev/staging environments, drop to one NAT per region (lose per-AZ failover for outbound traffic but save half the NAT bill). Or use VPC endpoints for S3/DynamoDB to bypass NAT for AWS-service traffic entirely.

Cross-region data transfer is not free. RDS replication traffic, S3 replication traffic, and any cross-region application calls are billed at $0.02/GB. A high-write database can quietly run up hundreds of dollars per month in cross-region transfer alone. Monitor the BytesReplicated metric on the RDS replica.

Provider injection in modules: explicit > implicit. When a module uses a non-default provider (like the assets module's aws.replica), the root config MUST pass it explicitly via the providers = { aws.replica = aws.west } block. Terraform will not infer this; omitting it produces a confusing "Provider configuration not present" error at plan time.

Cost expectations. A baseline two-region prod deploy runs roughly $300–500/month before any user traffic: ~$130 for NAT Gateways, ~$60 for two ALBs, ~$80–150 for two db.t3.medium RDS instances (Multi-AZ on the primary doubles its cost), ~$30 for EC2 instances, $1–2 for Route 53, plus cross-region transfer. The architecture is genuinely production-shaped — and so are the bills.

Where I Am At

Day 27 is the capstone of the infrastructure track: a real two-region active-passive deployment with Route 53-driven DNS failover, RDS cross-region replication, and S3 cross-region replication for static assets. Every piece — VPC, web tier, database, DNS, assets — is a Terraform module, parameterized by region, composed once at the root with two provider aliases.

The architecture is honest about its trade-offs. It's active-passive, not active-active, because cross-region synchronous writes are a different (much harder) problem. The database failover is asynchronous, so promotion costs you whatever was in flight. The bills are real because cross-region redundancy genuinely costs more — NAT Gateways alone are ~$130/month before anything else. None of that is a flaw of the design; it's the shape of the problem.

What's been built across Days 9–27 is a complete production-ready Terraform pattern: small modules, remote state with locking, environment separation, IMDSv2 hardening everywhere, security-group references between tiers, scaling owned by AWS, and now multi-region failover.

This post is part of a 30-day Terraform learning journey.