Debugging AWS Services Integration
Chunking a Multi-Tenant ECS Provisioning Pipeline for Reliability
A business automation platform provisions a full stack of 13 ECS microservices for every new tenant: service discovery, load balancer routing for the two public-facing services, the ECS services themselves, and autoscaling. The existing script did all of this in one Lambda invocation, so a single service failing partway through aborted the whole tenant's setup. I traced the script, found the failure mode, and split it into two stages, one that bootstraps service discovery and divides the work into parallel chunks, and one that provisions each chunk and keeps going past a single failure.
The problem
The client runs a multi-tenant SaaS platform where onboarding a new company means standing up its own copy of 13 backend services on ECS, wired together with private DNS and, for the two public services, routed through a shared load balancer. The provisioning script that did this ran everything for a tenant, all 13 services, their target groups, their autoscaling policies, in a single Lambda invocation, one service after another. If any one of the 13 hit an AWS error, an already-taken listener priority, a bad task definition, a throttled API call, the whole run stopped there. Every service after the failure was never created, and the tenant was left half-provisioned with no clear record of which services had actually gone through.
Constraints
- ·Existing script, not a rewrite: the fix had to keep the same AWS resources and task-definition format already in use.
- ·Thirteen interdependent services per tenant, two public-facing (frontend, backend) needing load balancer routing, eleven internal ones needing only service discovery.
- ·Had to work inside Lambda's execution model, not as a long-running process.
- ·Sole engineer: traced and fixed without a team to hand pieces to.
Architecture
Stage 1 (a Lambda) creates a per-tenant private DNS namespace, waits for it to finish creating, registers 12 CloudMap services under it, then builds one job description per service (13 total, including frontend and backend) and splits them into 5 near-equal chunks. Stage 2 (the same Lambda function, run once per chunk) downloads each service's task definition from S3, registers it with ECS, creates an ALB target group and routing rule only for frontend and backend, creates or updates the ECS service with binpack placement and a deployment circuit breaker, then registers Application Auto Scaling (CPU 80%, memory 60%) and a CloudWatch alarm per metric. A failed service inside a chunk is logged and skipped rather than aborting the chunk.
- ›Stage 1 Lambda (1.py): CloudMap namespace + 12 service registrations, then chunk builder (13 services into 5 groups)
- ›Stage 2 Lambda (2.py): per-chunk provisioning, run once per chunk, tolerates a single service failing
- ›ALB target groups and listener rules for frontend (HTTP, host-header) and backend (HTTPS, host-header plus /api/*, /libraries/*, /socket/* path patterns) only
- ›CloudMap private DNS namespace and per-service registration for the other 11 internal services
- ›Application Auto Scaling target (1-3 tasks) and 2 CloudWatch alarms (CPU, memory) per service
- ›create_ecs_service.py: the retired single-invocation version kept for reference, not deployed
- ›lambda.py and 1v1.py: a local payload-shape check and an incomplete draft, neither deployed
How one request flows
- 01Bootstrap service discovery: stage 1 creates a private DNS namespace named for the tenant, waits for it to finish creating, then registers 12 CloudMap services under it, one per internal microservice.
- 02Build the work list: stage 1 assembles one job description per service (its S3 task-definition key, security group, CloudMap id) for all 13 services, frontend and backend included.
- 03Split into 5 chunks: the 13 jobs are divided into 5 near-equal groups and returned as the stage's output, ready for 5 parallel runs of stage 2.
- 04Provision each service in a chunk: stage 2 downloads each service's task definition from S3, registers it with ECS, and for frontend or backend only, creates an ALB target group and its routing rule.
- 05Create the ECS service: each service is created or updated with binpack placement, a deployment circuit breaker, and, where applicable, its target group and CloudMap registration wired in.
- 06Attach autoscaling: every service gets an Application Auto Scaling target, a CPU policy at 80%, a memory policy at 60%, and one CloudWatch alarm per metric.
- 07Keep going past a failure: if one service in a chunk fails, stage 2 logs it and moves to the next service in that chunk instead of stopping the chunk.
Engineering decisions
Splitting one long run into parallel chunks
What. Split the original single-invocation script into a bootstrap-and-chunk stage and a separate per-chunk provisioning stage, so the 13 services run as 5 parallel units instead of one long sequential chain.
Why. Bounds how much of a tenant's setup a single bad service can take down.
Tradeoff. A second deployable and an extra orchestration step (a Step Functions map or five direct invocations) instead of one simple function.
One failing service should not fail the other 12
What. The per-chunk stage catches a failed service, logs it, and continues to the next one in the chunk.
Why. The original script returned immediately on the first error, so a bad task definition or a transient AWS error partway through aborted every service after it.
Tradeoff. A chunk can now report success while one of its services quietly failed. There is no retry or re-queue for that service yet, so its output still needs checking, not just trusting.
Only two of thirteen services need load balancer routing
What. Only frontend and backend get an ALB target group and routing rules, frontend by host-header, backend by host-header plus three path patterns for its API, library, and socket routes.
Why. The other 11 services are internal and reached by name over the tenant's private DNS namespace, which keeps the ALB's listener rule count from growing with every internal service a tenant has.
A dedicated DNS namespace per tenant
What. Each tenant gets its own private DNS namespace and a CloudMap service entry per internal microservice, created before any ECS service exists.
Why. Namespace creation in AWS is asynchronous, so the script polls the operation status until it completes rather than assuming success. This gives the 13 services a name-based way to reach each other that survives ECS tasks restarting and getting new IPs.
A fixed chunk count instead of a derived one
What. The number of chunks is a literal 5 in the code, not calculated from how many services a tenant actually has.
Why. For the current fixed shape of 13 services this is simple and predictable.
Tradeoff. A real limitation, not a strength. If the per-tenant service count ever changes, the chunk count has to be revisited by hand.
What changed along the way
- →The original single-invocation script worked when every service succeeded, but any one failure partway through aborted the whole tenant, with no record of which services had actually completed.
- →The fix split the work into a setup-and-chunk stage and a per-chunk provisioning stage, so a failure is contained to one service instead of the whole run.
- →The chunk count stayed a fixed 5 rather than being derived from the tenant's actual service count, a limitation carried into the fix rather than solved by it.
Security and reliability
- ·Private networking: ECS services run in private subnets with no public IP assigned.
- ·Scoped routing: ALB rules match on host-header and path-pattern per tenant, so one tenant's traffic cannot reach another's services.
- ·Deployment circuit breaker enabled on every ECS service update, though automatic rollback is off, a limitation worth flagging, not a strength.
- ·A CloudWatch alarm on CPU and one on memory for every service, not a shared threshold across the tenant.
- ·Failure isolation: the per-chunk stage continues past a single failed service instead of aborting the batch.
Metrics
What shipped
- ✓Traced and corrected the tenant provisioning pipeline: CloudMap namespace and service registration, ALB target groups and routing rules for the two public services, ECS service creation, and per-service autoscaling with CloudWatch alarms.
- ✓Split the original single-invocation script into a two-stage pipeline, bootstrap and chunk, then per-chunk provisioning, so one bad service no longer takes the other 12 down with it.
- ✓Replaced the hardcoded tenant domain and test-fixture AWS identifiers with environment-configured values.
- ✓Flagged, rather than papered over, the fix's own limitations: a fixed chunk count, no rollback on the circuit breaker, and no retry for a service that failed inside a chunk.
Lessons learned
- ·A single long Lambda invocation doing many independent AWS operations concentrates failure, one bad step aborts everything after it. Chunking bounds the blast radius.
- ·Catching and logging a per-item failure inside a loop, instead of letting it raise, is often the simplest way to turn an all-or-nothing batch job into a mostly-succeeds one.
- ·Asynchronous AWS operations, such as CloudMap namespace creation, need explicit status polling, not a create-and-continue call.
- ·A fixed chunk count is a reasonable first fix, but it stays a real limitation until it is derived from the actual workload size.