Enterprise EKS Infrastructure & Full-Stack Deployment Platform
An Enterprise EKS Platform for a Multi-Service Web Application
An enterprise software company needed its frontend, backend API, and background job processor running on a production Kubernetes platform instead of a handful of manually managed servers. I built the infrastructure as reusable Terraform modules, VPC, EKS, RDS, and Redis, and deployed all three services onto the cluster, each with its own Kubernetes Deployment, Service, ALB Ingress, and security group. The result is a modular, service-isolated platform where any one part can be reworked without touching the others.
The problem
The client ran a multi-part application, a web frontend, a backend API, and a Hangfire background job processor, that needed a shared, production-grade home instead of ad hoc hosting. Each of the three parts has different scaling and availability needs: the frontend and API take user traffic, Hangfire runs scheduled and queued jobs. Running all three off one server or one undifferentiated deployment means a spike or a bad release in one takes down the others, and there is no way to scale one independently. They needed each service isolated at the network and deployment level, backed by a managed database and cache, provisioned in a way that could be rebuilt and audited as code.
Constraints
- ·Kubernetes-based: the client had standardized on AWS EKS for the target platform.
- ·SQL Server as the database engine, which points at a .NET-based application stack.
- ·Hangfire as the background job processor, deployed as its own scaled Kubernetes service rather than in-process.
- ·Single engineer: infrastructure, manifests, and module design all owned end to end.
- ·No CI/CD pipeline in scope: deployment is direct Terraform and kubectl against ECR-hosted images.
Architecture
A VPC module carves public and private subnets across the region's AZs with one NAT gateway per public subnet. An EKS module, built on the community terraform-aws-modules/eks/aws module, provisions the managed control plane and a single on-demand node group into the private subnets, with the eks-blueprints-addons module installing VPC CNI, kube-proxy, CoreDNS, the EBS CSI driver, the AWS Load Balancer Controller, and the cluster autoscaler. The EBS CSI driver gets its own IRSA role instead of node-wide permissions. Three Kubernetes manifest sets, frontend, backend, and hangfire, each define a Deployment (one container pulling from ECR), a ClusterIP Service, and an internet-facing ALB Ingress, with each Deployment pinned to its own security group via the pod-ENI annotation. Standalone RDS (SQL Server) and ElastiCache (Redis) modules attach to the same VPC for shared state.
- ›modules/vpc: VPC, internet gateway, public/private subnets, NAT gateways, route tables
- ›modules/eks: EKS cluster (terraform-aws-modules/eks/aws), one managed node group, eks-blueprints-addons (VPC CNI, kube-proxy, CoreDNS, EBS CSI, ALB controller, cluster autoscaler), IRSA role for the EBS CSI driver
- ›modules/redis: single-shard ElastiCache Redis replication group with its own security group
- ›modules/rds: SQL Server RDS instance (gp3, 100 GB) with its own security group and DB subnet group
- ›yaml/frontend, yaml/backend, yaml/hangfire: Deployment + ClusterIP Service + ALB Ingress per service, each with a dedicated security group annotation
How one request flows
- 01A client request hits one of three internet-facing ALB Ingresses (frontend, backend, or hangfire's dashboard/API), each provisioned from its own Ingress manifest.
- 02The ALB Controller, running as a cluster addon, routes the request to the matching ClusterIP Service on port 8080.
- 03The Service forwards to the Deployment's pod, running the image pulled from that service's ECR repository, inside the EKS managed node group in the private subnets.
- 04The pod's ENI carries a security group scoped to that one service, via the k8s.amazonaws.com/sg-ids annotation, not the shared node security group.
- 05Backend and Hangfire reach the SQL Server RDS instance and the ElastiCache Redis replication group over the VPC's private networking for persistent state and caching.
- 06Hangfire's own Deployment processes background and scheduled jobs independently of frontend and backend traffic, scaling and failing separately from the user-facing services.
- 07Persistent volumes, where used, are provisioned through the EBS CSI driver, whose AWS permissions come from an IRSA role bound through the cluster's OIDC provider.
Engineering decisions
One security group per service, not one shared node group
What. Each Deployment (frontend, backend, hangfire) sets its own security group ID via the k8s.amazonaws.com/sg-ids pod annotation.
Why. A shared node security group means every pod on a node has the same network exposure. Pinning a security group per service keeps a change to one service's network rules from touching the others.
Tradeoff. One more security group to manage per service, chosen over the simpler but coarser option of a single cluster-wide security group.
Hangfire deployed as its own service, not in-process
What. Hangfire gets the same Deployment/Service/Ingress shape as frontend and backend, its own image, and its own ALB ingress.
Why. Background job load and web traffic have different scaling profiles. Running Hangfire as its own Kubernetes service lets it scale, restart, or fail without affecting the user-facing frontend and backend.
Tradeoff. A third full manifest set and ingress to maintain, instead of running Hangfire inside the backend process.
IRSA for the EBS CSI driver, not node-wide IAM
What. The EBS CSI driver's AWS permissions come from an IAM role bound to its Kubernetes service account through the cluster's OIDC provider.
Why. Attaching the same permissions to the node's IAM role would hand every pod on that node the same storage access. IRSA scopes it to the one service account that needs it.
Tradeoff. None significant: this is the standard EKS pattern for addon permissions and adds no real operational cost.
Modular Terraform, one module per resource type
What. VPC, EKS, RDS, and Redis are each their own Terraform module, composed from a single prod root module.
Why. Splitting by resource type makes each piece independently reviewable and reusable if a second environment is ever needed.
Tradeoff. The module boundary has to be kept in sync by hand. As found during this cleanup, the RDS module's public_subnets reference was never wired through the root module, a real cost of splitting state across module boundaries without an interface check.
Managed EKS add-ons over hand-rolled controllers
What. The AWS Load Balancer Controller, cluster autoscaler, and CNI/DNS components are installed through the community eks-blueprints-addons module rather than applied as raw manifests.
Why. These are well-maintained upstream components, and wiring them through a maintained module is less code to own and keeps them aligned with the EKS version in use.
Tradeoff. Less direct control over exact addon versions and config than hand-writing the manifests, accepted for lower maintenance burden.
What changed along the way
- →The RDS module was written to take private_subnets like every other module in this stack, but modules/rds/main.tf still references an undeclared public_subnets variable for the DB subnet group. This surfaced during this cleanup pass and needs a fix (either declare and pass public_subnets, or switch the DB subnet group to private_subnets) before the stack can plan cleanly again.
- →RDS and Redis security groups currently allow inbound from 0.0.0.0/0 on their database and cache ports rather than being scoped to the VPC CIDR or the node security group. This is a fast-path pattern for standing the stack up, not a hardened default, and is called out here rather than carried forward silently.
Security and reliability
- ·Scoped identity: the EBS CSI driver's AWS permissions come from an IRSA role bound through OIDC, not the node's IAM role.
- ·Network isolation per service: each of frontend, backend, and hangfire gets its own security group at the pod level.
- ·Private placement: the EKS node group, RDS instance, and Redis replication group all sit in the VPC's private subnets, reached only through the ALB ingresses on the public side.
- ·Known gap: RDS and Redis security groups currently allow inbound from 0.0.0.0/0 on their ports rather than a VPC-scoped range, flagged for tightening before production traffic.
- ·Known gap: the RDS module's subnet variable mismatch (see What changed) means this exact module version has not been shown to apply cleanly.
Metrics
What shipped
- ✓Delivered a modular Terraform stack provisioning VPC, EKS, RDS, and Redis for a production environment.
- ✓Deployed three independently scalable Kubernetes services, frontend, backend, and Hangfire, each behind its own ALB ingress and security group.
- ✓Wired IRSA for the EBS CSI driver instead of granting broad node-level permissions.
- ✓Configured the AWS Load Balancer Controller and cluster autoscaler as managed EKS add-ons.
- ✓Surfaced and documented a subnet-variable mismatch in the RDS module and an overly broad database security group, left as follow-up items rather than hidden.
Lessons learned
- ·Give every service its own security group at the pod level when isolation matters, not the shared node security group.
- ·Reach for IRSA and OIDC over node-wide IAM roles for any add-on that touches AWS APIs.
- ·Splitting Terraform into one module per resource type only pays off if the variables passed across the module boundary are kept in sync, an unpassed variable can sit unnoticed until the next apply.
- ·Wide-open security group ingress on a database or cache port is a fast way to get an environment running, not a production default, tighten it before real traffic arrives.