Data Lake Foundation on S3 + Lake Formation (Terraform)
A Governed Data Lake Foundation on AWS
A consultancy's data team needed a governed landing zone before they could start building anything on top of it: a lake with clear storage tiers, centralized access control, and a repeatable way to provision and tear it down. I built the foundation as Terraform: a medallion-layout S3 lake registered with AWS Lake Formation for tag-based governance, per-layer Glue IAM roles ready for the promotion jobs, and a CodeBuild pipeline that plans, applies and destroys the whole stack against locked remote state. The ETL promotion path and a Snowflake integration were designed and written, then deliberately staged rather than switched on, so the next phase has a clear, working starting point instead of an empty repo.
The problem
The client is a consultancy standing up a new data platform for internal analytics. Before any pipeline work could start, someone had to decide the storage layout, wire up access control that would scale past a handful of manual bucket policies, and make the whole thing reproducible so a second environment was not a repeat of tribal knowledge. The naive path, clicking together a few S3 buckets and IAM policies by hand in the console, does not survive a second environment or a security review: permissions live nowhere in particular, storage tiers are informal, and nothing rebuilds from a clean account. What they needed was a lake laid out in clear stages (raw, conformed, modelled), governed centrally rather than bucket by bucket, and provisioned as code so it could be planned, reviewed, and applied the same way every time.
Constraints
- ·Foundation-first scope: this phase was the lake and its governance, not the ETL jobs that would run on top of it.
- ·AWS-native governance was required, so access control had to go through Lake Formation rather than per-bucket IAM policies alone.
- ·Everything had to be reproducible from code through a CI pipeline, not applied by hand from a workstation.
- ·A Lake Formation permissions grant cannot reference a Glue catalog that does not exist yet, which forced an explicit build order.
- ·Built by a single engineer, so scope had to stay tight enough to land a working foundation in the time available.
Architecture
A root Terraform module composes three live sub-modules. S3 provisions a base bucket plus raw, conformed and modelled buckets with private ACLs. Lake Formation registers the base bucket as a governed resource, sets the data lake administrator from the caller's own IAM session rather than a hardcoded ARN, and defines an LF-tag for tag-based access control. Glue provisions the per-layer IAM roles and policies that a promotion job would assume. A fourth sub-module, the Glue crawler/job/trigger promotion path, is written in full but left block-commented, and a fifth, a Snowflake automation module, is written in full but commented out of the root. The whole stack is planned, applied and destroyed through three separate CodeBuild projects against remote state locked in DynamoDB.
- ›Terraform root module: composes S3, Glue and LakeFormation, the Snowflake module exists in source but is not called
- ›S3 module: base, raw, conformed and modelled buckets, private ACLs, bucket names passed out as module outputs
- ›Lake Formation module: resource registration, session-derived admin, one LF-tag, the permissions grant is written but commented out pending the Glue catalog
- ›Glue module: per-layer IAM roles and policies are live, the catalog, crawlers, jobs and ON_DEMAND triggers are written but block-commented
- ›Snowflake module: six sub-modules (database, schema, roles, user, warehouse, grants) using the Snowflake-Labs provider plus a second provider for grants it cannot express, complete but not wired into the root
- ›CodeBuild: three buildspecs (plan, apply, destroy) each installing Terraform and running init before the corresponding action
- ›Remote state: an encrypted S3 backend with a DynamoDB lock table
How one request flows
- 01A change lands in the Terraform source: a module edit, a new variable, or (in the next phase) uncommenting the Glue promotion path.
- 02CodeBuild's plan project installs Terraform, runs terraform init against the locked S3 backend, and runs terraform plan so the diff is visible before anything changes.
- 03A reviewed plan is applied by the separate apply project, never by the plan project itself, so a plan review can never accidentally become a destroy.
- 04S3 provisions the four buckets and hands their names out as outputs, so Glue and Lake Formation never hardcode a bucket string.
- 05Lake Formation registers the base bucket, derives the admin from the caller's own session, and creates the LF-tag used for tag-based access.
- 06Glue provisions the per-layer IAM roles ready to be assumed once the crawlers and jobs are enabled.
- 07If the stack needs to come down, the separate destroy project runs terraform destroy against the same locked state, kept intentionally apart from apply.
Engineering decisions
Medallion layout as separate buckets, not prefixes
What. Raw, conformed and modelled data each get their own S3 bucket rather than three prefixes inside one bucket.
Why. Separate buckets give each tier its own IAM boundary, lifecycle rules and Lake Formation registration without one bucket policy trying to express three different trust levels at once.
Tradeoff. More buckets to register and govern individually, accepted because tier-level isolation was the point.
Lake Formation over per-bucket IAM
What. Registered the lake as a Lake Formation resource and governs it with an LF-tag instead of relying on S3 bucket policies alone.
Why. Tag-based access control scales as tables and consumers are added later. Per-bucket IAM policies would need hand-editing for every new principal.
Tradeoff. A second permissions layer to reason about on top of IAM, worth it for how access control was meant to grow.
Data lake admin derived from the caller's session, not hardcoded
What. The Lake Formation admin comes from `data.aws_caller_identity` and `data.aws_iam_session_context` at apply time.
Why. A hardcoded admin ARN breaks the moment the identity applying Terraform changes, a different engineer, a different CI role. Deriving it keeps the module portable across who runs it.
Tradeoff. None significant, this is strictly safer than the hardcoded alternative.
Destroy as its own pipeline, not a flag
What. Plan, apply and destroy are three separate CodeBuild projects, each with its own buildspec.
Why. A single buildspec branching on a variable means one bad input or one careless run could destroy instead of apply. Separating them removes that failure mode entirely.
Tradeoff. Three buildspecs to maintain instead of one, accepted for what it removes from the blast radius.
Staging the Glue promotion path instead of shipping it half-tested
What. Wrote the full catalog, crawler, job and trigger definitions, then left them block-commented rather than applying them in this phase.
Why. The permissions grant that would let those jobs write into the catalog cannot exist before the catalog does, and this phase's scope was the governed foundation, not the promotion jobs. Shipping a working foundation beats shipping a half-wired pipeline.
Tradeoff. The promotion path is not live yet. That is stated plainly rather than presented as delivered.
Snowflake module built ahead, not bolted on later
What. Wrote a complete Snowflake automation module (database, schema, roles, users, warehouse, grants) and left it commented out of the root module.
Why. Building it against the same variable and module conventions as the rest of the stack now means turning it on later is one line in main.tf, not a rewrite.
Tradeoff. Unused code sitting in the repo until it is enabled, accepted because the alternative was building it from scratch in a later phase.
What changed along the way
- →Started wiring the Lake Formation permissions grant directly, then found it referenced a Glue catalog that would not exist until the (also staged) Glue module runs, so the grant was left commented with the ordering constraint written next to it instead of forced through.
- →The Snowflake module ended up fully built during this phase rather than deferred, once it became clear the same variable and provider-alias conventions used elsewhere in the stack would carry over cleanly.
- →Moved Snowflake's DCL grants (the ones the Snowflake-Labs provider itself cannot express) onto a second, purpose-built provider rather than working around the gap with local-exec.
Security and reliability
- ·Every S3 bucket is private (explicit private ACL on all four buckets), nothing public by default.
- ·Lake Formation governs access centrally through an LF-tag rather than bucket-by-bucket IAM policies.
- ·The data lake administrator is derived from the caller's IAM session at apply time, never a hardcoded identity.
- ·Remote state is encrypted in S3 and locked via DynamoDB, so concurrent applies cannot corrupt state.
- ·Destroy is a separate CodeBuild pipeline from apply, so a destructive action requires a distinct, deliberate trigger.
Metrics
What shipped
- ✓Medallion-layout S3 lake (raw / conformed / modelled) provisioned as code with private ACLs throughout.
- ✓Lake Formation registration, session-derived admin, and an LF-tag in place for tag-based governance.
- ✓Per-layer Glue IAM roles and policies applied, the crawler/job/trigger definitions written and ready to enable.
- ✓Terraform-over-CodeBuild pipeline with locked remote state and a separated destroy path.
- ✓A complete Snowflake automation module built ahead of need, ready to switch on with a single module block.
Lessons learned
- ·A governed foundation that stages its next phase cleanly (commented, not deleted, with the ordering constraint documented) is worth more than rushing a half-tested pipeline into a first apply.
- ·Deriving identity from the caller's session instead of hardcoding an ARN is a small change that makes a module portable across whoever runs it.
- ·Splitting plan, apply and destroy into separate pipelines removes an entire class of accidental-destroy risk for the cost of one extra buildspec.
- ·Building a dependent module (Snowflake) against the same conventions as the rest of the stack, even before it is switched on, keeps the eventual integration to one line instead of a rewrite.