AWS Redshift Data Warehouse (Terraform, Scaffold)
Redshift Data Warehouse Scaffold
A company wanted to see the shape of a Redshift-based data warehouse in Terraform before committing engineering time to a full build. I wrote two focused modules, a single-node Redshift cluster and a Terraform Cloud workspace to run it through, and a root file meant to compose them behind a VPC. This is early-stage scaffold code. It sketches the right pieces of a warehouse but was never finished into something that plans or applies, and that gap is stated plainly here rather than hidden.
The problem
The client needed a Redshift data warehouse: a cluster to hold analytics data, a repeatable way to provision it, and a Terraform Cloud workspace so applies would run through a managed remote backend rather than a laptop. Nothing existed yet beyond the intent to use Redshift as the warehouse engine. Clicking a cluster together by hand in the console does not survive a second environment. What the client needed first was the shape of the thing in code, a cluster module with its own variables and a workspace to run it through, so the pattern could be reviewed and reused rather than rebuilt from scratch each time.
Constraints
- ·Very early stage: the redshift module was written first, four days before the workspace module and the root file. Neither round of work finished the wiring between them.
- ·No VPC module ever written: the root file assumes a VPC module supplies the subnet group and security group the Redshift module needs, but that module was never created.
- ·No Terraform Cloud credentials wired in: the workspace module needs an organization and an API token to authenticate, neither was supplied.
- ·Single engineer, no review pass: built solo and never run through terraform init, so path and naming mistakes were never caught before this audit.
Architecture
Two real Terraform modules exist on disk, redshift (a single-node aws_redshift_cluster) and workspace (a Terraform Cloud tfe_workspace), plus a root main.tf that declares three module blocks meant to compose a VPC, the Redshift cluster, and the workspace. None of the three source paths in the root file resolve to the real directories, so the composition was never actually wired.
- ›Root main.tf: three module blocks (vpc, redshift, example_workspace), all three source paths unresolved
- ›module/redshift: aws_redshift_cluster "default", single-node dc2.large, parameterized identifier/database/credentials/node type
- ›module/workspace: tfe_workspace resource, parameterized name/organization/description/auto_apply, tfe provider commented out
- ›vpc module: referenced by the root file and by the redshift module's depends_on, never written
How one request flows
- 01terraform init would run first. As committed, it fails immediately: none of the three module source paths in main.tf resolve to a real directory.
- 02module.vpc was meant to build the network first, supplying the subnet group and security group the Redshift module takes as inputs. This module was never written.
- 03module.redshift was meant to consume the VPC's outputs. Its depends_on lists a VPC, a security group, a subnet group and an IAM role, none of which exist inside this module.
- 04The cluster itself is fully declared: a single-node dc2.large Redshift cluster with its own identifier, database name, and master credentials as variables.
- 05The cluster's own output would fail to resolve: output.tf reads a resource named redshift_cluster, but the resource in main.tf is named default.
- 06module.example_workspace was meant to register a Terraform Cloud workspace. Its source path also does not match the real module directory, and the tfe provider it needs is commented out.
- 07No apply was ever attempted. There is no state file, no CI pipeline, and no plan output anywhere in the project.
Engineering decisions
Redshift and workspace concerns kept in separate modules
What. The cluster and the Terraform Cloud workspace live in their own module directories rather than one flat file.
Why. Either module can be fixed and finished on its own without the other blocking it, the right instinct at scaffold stage.
Tradeoff. That separation only pays off once the wiring between the modules and the root actually works, which it does not yet.
Cluster configuration parameterized from day one
What. Identifier, database name, master credentials, node type and cluster type are all variables with defaults rather than hardcoded into the resource block.
Why. That is the shape you want before standing up a second environment.
Tradeoff. No variable validation and no non-default node type were set, both reasonable next steps once the module actually plans.
The root file's three module paths do not match the real layout
What. main.tf reaches for ./vpc, ./redshift and ./workspace-module. Only two of those modules exist at all, and both live under module/, not at the paths the root file names.
Why. Named directly rather than smoothed over: as committed, terraform init cannot get past resolving these three lines.
Tradeoff. None, this is a defect to fix, not a design choice.
The workspace module's variables confuse type with default
What. Every variable in module/workspace/variable.tf writes type = "<a literal value>", for example type = "<a name string>", where Terraform expects a type constraint such as string and a separate default argument for the value.
Why. This is invalid HCL, not a style issue, and would fail terraform validate on its own even if every path problem above were fixed.
Tradeoff. None, this needs a straight fix before the module can be planned.
The Redshift module's depends_on reaches for resources it doesn't own
What. The cluster's depends_on lists a VPC, a security group, a subnet group and an IAM role, written as if those resources were declared inside this module. None of them are.
Why. The instinct, make the cluster wait on its networking, was right. The execution assumed a dependency that does not exist yet, the VPC module that was never built.
Tradeoff. None, this is a defect, named directly rather than hidden.
What changed along the way
- →The Redshift module was written first, on its own, four days before the workspace module and the root composition file that tries to wire them together.
- →The root file's three module blocks were added in that second session without updating the paths to match where the redshift and workspace modules actually live under module/.
- →The VPC module the root file and the Redshift module's depends_on both assume was never started.
- →The Terraform Cloud provider block needed to authenticate the workspace module was written, then commented out, and never finished with real organization and token values.
Security and reliability
- ·No live secrets in the scaffold: the Redshift master username and password were literal defaults in source, sanitized to placeholders during this cleanup, with the real values moved to a gitignored .env.
- ·No remote state, no CI, no apply: there is no Terraform backend configuration, state file, or pipeline anywhere in the project.
- ·skip_final_snapshot is set to true, reasonable for a scaffold that was never going to hold real data, worth revisiting before any real environment is built from this module.
- ·No IAM role attached to the cluster: the iam_roles argument is commented out, so the cluster as written has no path to read from S3 or other AWS services.
Metrics
What shipped
- ✓A parameterized single-node Redshift cluster module: identifier, database name, credentials, node type and cluster type all exposed as variables.
- ✓A Terraform Cloud workspace module sketching name, organization, description and auto-apply behavior.
- ✓A root file expressing the intended composition, VPC in front of Redshift, plus a separate workspace, even though none of its three paths resolve as written.
- ✓A full, named list of what would need fixing before this validates: three broken module paths, one broken output reference, one depends_on pointing at undeclared resources, and invalid variable syntax across the workspace module.
- ✓Credentials sanitized out of the module defaults during this cleanup pass, with the real values moved to a gitignored .env.
Lessons learned
- ·A module's own output file has to be checked against the resource name actually declared in that module, not the name it started with or was meant to have.
- ·A root file's module source paths need to be verified against the real directory layout the moment either one changes, a rename on one side with no matching update on the other is exactly how this broke.
- ·Terraform's type argument on a variable is a type constraint, not a place to put the default value, and nothing catches that mistake until terraform validate actually runs.
- ·Naming a scaffold's gaps plainly, three broken paths, one broken output, one dangling depends_on, is worth more than presenting early work as further along than it is.