Multi-Environment AWS Infrastructure (ECS, Redis, Client VPN)
Parallel Production and Staging Infrastructure on AWS
A consumer web platform needed a production environment and a staging environment that could never touch each other by accident, plus a way for operators to reach internal services without opening anything to the public internet. I built the whole stack in Terraform: two parallel AWS environments, each with its own VPC, load balancer, Fargate service, and Multi-AZ Redis cluster, reachable for operators only through a certificate-authenticated Client VPN.
The problem
The platform's team needed staging to be a real rehearsal of production, not a shared environment with a different tag. A shared VPC or a shared cache between the two meant a staging mistake, a bad migration, a runaway job, could reach production data or production capacity. At the same time, the application depends on Redis for caching and session state, and that Redis instance had no reason to be reachable from the internet. But someone still had to be able to connect to it to debug a problem or check its state by hand. A public endpoint was not an option, and a shared always-on bastion host was more infrastructure to patch and pay for than the problem justified.
Constraints
- ·Two-month build, one engineer end to end.
- ·Full isolation between production and staging: no shared network, cache, or compute cluster.
- ·No public path to internal services (Redis, internal ECS ports).
- ·Cost-conscious compute: Fargate tasks default to zero running.
- ·No existing PKI or VPN infrastructure to build on.
Architecture
Two fully separate AWS environments (production, staging), each declared as its own module tree: a VPC with public/private subnets and per-AZ NAT Gateways, an internet-facing ALB (HTTP redirected to HTTPS, TLS 1.3 via a certificate shared by both environments), a Fargate ECS service in the private subnets with CPU-based autoscaling from zero, and a Multi-AZ ElastiCache Redis replication group locked to VPC-internal access. A per-environment AWS Client VPN endpoint, authenticated against a self-managed Easy-RSA certificate authority, is the only path into the private subnets. Amazon SES handles outbound mail with a DNS-verified sending domain.
- ›VPC (public/private subnets, per-AZ NAT Gateway) x2
- ›Application Load Balancer (HTTP->HTTPS, ACM TLS 1.3) x2
- ›ECS Fargate service (CPU autoscaling 0-2 tasks) x2
- ›ElastiCache for Redis (Multi-AZ, redis7, automatic failover) x2
- ›AWS Client VPN endpoint + self-managed Easy-RSA CA x2
- ›Amazon SES (DKIM/TXT DNS-verified domain, shared)
How one request flows
- 01The request reaches the ALB. An HTTP listener on port 80 redirects to HTTPS. The HTTPS listener on 443 terminates TLS 1.3 using the ACM certificate shared by both environments.
- 02The ALB forwards to a healthy Fargate task. The target group health-checks /healthz before sending traffic, and the task runs in a private subnet with no public IP.
- 03The task talks to Redis if it needs cached state. The Redis security group only accepts connections from the ECS security group and from inside the VPC, never from the internet.
- 04CloudWatch watches CPU. If average CPU crosses 80 percent, an appautoscaling target-tracking policy scales the ECS service out, up to two tasks. Below that, it can scale back to zero.
- 05An operator who needs to check Redis by hand connects over Client VPN. Their client certificate, issued by that environment's own Easy-RSA CA, authenticates the connection, and from there they run redis-cli straight against the private Redis endpoint.
- 06Outbound mail leaves through SES. The sending domain is verified at the DNS layer with DKIM and a TXT ownership record.
Engineering decisions
Two parallel module trees instead of one shared, parameterized module
What. Production and staging are built as two separate module trees (vpc_prod/vpc_staging, alb_prod/alb_staging, and so on), each with its own resources and no shared state, instead of one module branching on an environment variable.
Why. Isolation becomes the shape of the code itself instead of a runtime branch that can be misconfigured. A staging change cannot reach production because there is no code path that lets it.
Tradeoff. It costs roughly twice the Terraform to write and to keep in sync by hand, versus the lower line count of one shared, parameterized module.
Certificate-based Client VPN instead of a bastion host or a public Redis endpoint
What. An AWS Client VPN endpoint per environment, authenticated with mutual TLS against a self-managed Easy-RSA certificate authority, is the only path into the private subnets.
Why. Redis needed an access path for operators without a public IP or an always-on box to patch. Connecting requires a signed client certificate, not a password, and there is nothing listening on the internet to attack.
Tradeoff. A bastion host would have been faster to stand up initially but is one more host to patch, size, and pay for continuously.
Self-managed Easy-RSA PKI instead of AWS Private CA
What. Each environment's Client VPN certificate authority runs on Easy-RSA, issuing a handful of certificates locally rather than through a managed service.
Why. AWS Private CA bills monthly regardless of issuance volume, and this project needed exactly two CAs issuing a small, fixed set of certificates.
Tradeoff. Manual key management and rotation instead of a managed service. The right call at this scale, the wrong one past it.
Fargate over EC2-backed ECS
What. Both ECS services run on Fargate rather than an EC2-backed cluster.
Why. No host to patch, size, or scale by hand, at the cost of paying per task rather than per reserved instance.
Tradeoff. Less control over the underlying compute, accepted because the workload spends real time at zero desired count where per-task pricing wins.
Scaling from zero by default
What. Both ECS services default to desired_count = 0, with autoscaling capped at 2 tasks on CPU pressure.
Why. Neither environment needed to run around the clock, and an idle environment sitting at a nonzero task count is a cost nobody notices until the bill arrives.
Tradeoff. A cold start the first time traffic returns after scaling to zero, accepted for this workload.
One ACM certificate shared across both environments' ALBs
What. Both HTTPS listeners reference the identical ACM certificate ARN rather than each environment provisioning its own.
Why. Cuts certificate management to one place instead of two.
Tradeoff. Production and staging now share a trust boundary on that certificate, so revoking or rotating it is a coordinated action across both environments, not an isolated one.
What changed along the way
- →The first task definitions shipped without environment variables baked in, and had to be revisited once it was clear the container needed per-environment configuration at runtime.
- →A later cleanup pass found leftover resources (an old VPC, an ECS namespace, a security group, a couple of public IPs) never torn down after earlier iterations, and marked them for deletion rather than leaving them running.
- →Redis naming went through a few iterations (production, development, then a plain cluster name) before settling on the current per-environment scheme, evidence the isolation boundary was worked out in practice, not just on paper.
Security and reliability
- ·No public path to internal services: ECS tasks and Redis sit in private subnets with no public IP. The only route in for an operator is the certificate-authenticated Client VPN.
- ·Security groups scoped by relationship, not by CIDR: the ECS security group only accepts traffic from the ALB security group, and Redis only accepts traffic from the ECS security group or the VPC CIDR.
- ·TLS everywhere on the public edge: both ALBs redirect HTTP to HTTPS and terminate on TLS 1.3.
- ·Mutual TLS on the VPN, not a shared secret: each environment has its own CA, so a compromised staging certificate has no path to production.
- ·Least-privilege task role: ECS tasks assume a role scoped to the standard AmazonECSTaskExecutionRolePolicy, not a broad account-wide grant.
- ·CloudWatch Container Insights enabled on both ECS clusters for visibility into task health and resource use.
Metrics
What shipped
- ✓A working Terraform stack that stands up two independent AWS environments (production and staging) for the the platform application from a single root module.
- ✓Each environment runs its container on Fargate behind an HTTPS ALB with TLS termination and a health-checked target group.
- ✓Each environment has its own Multi-AZ Redis replication group, reachable only from inside its own VPC.
- ✓Operators can reach private-subnet resources in either environment through a certificate-authenticated Client VPN connection instead of any public exposure.
- ✓Outbound email sending is verified at the DNS level via SES DKIM/TXT records.
Lessons learned
- ·Two parallel module trees make isolation easy to reason about, but they are twice the Terraform to maintain, worth it at this scale and worth reconsidering above it.
- ·A certificate-based VPN is a durable, low-cost way to reach private resources, it does not need a bastion host or a public endpoint to work.
- ·Scaling ECS to zero by default catches an idle environment before it turns into a bill nobody explained.
- ·Security group rules added for convenience (an extra admin port, a hardcoded CIDR) need the same review as the main ingress rules, they age worse because nobody remembers why they are there.
- ·Verifying an email sending domain at the DNS layer before launch avoids fighting delivery limits later.