2023 · Data platform
A governed data lake foundation on AWS
a global management consultancy · Data engineer
The problem
A consultancy was starting a new data platform for internal analytics. Storage layout and access control had to be decided before any pipeline work.
What I built
I built the data lake as code: raw, cleaned and modelled storage layers with access rules defined from the start, and a pipeline to create it again.
The result
The analytics team could start building pipelines on a base with permissions already settled.
Main tools
- Terraform
- Amazon S3
- AWS Lake Formation
- AWS Glue
- AWS CodeBuild
This is the short version. The full case study is available on request.