Enterprise EKS MLOps Platform Development & Advanced Model Orchestration

The problem

A data team was doing machine learning without a shared platform. Training ran on people's machines, experiments were not tracked, and there was no repeatable path from a notebook to a managed model. They wanted the standard open-source tools, Kubeflow for pipelines and notebooks and MLflow for tracking and the registry, but running on their own infrastructure. A hosted SaaS was ruled out because the data had to stay in their AWS account. Installing these tools by hand is where teams get stuck: Kubeflow is a large bundle that needs persistent storage, an ingress, and authentication, all wired together. They needed it stood up in a repeatable way, not a one-off manual install nobody could reproduce.

Constraints

• ·Self-hosted in the client's own AWS account, so a managed ML service was ruled out.
• ·Kubernetes-based, because Kubeflow and MLflow run on Kubernetes and need storage, ingress, and auth.
• ·Reproducible: the cluster and platform had to rebuild from code, not be clicked together once.
• ·Built by a single engineer, so the moving parts had to stay manageable.

Architecture

Infrastructure as code (Terraform modules for VPC, EKS, and a bastion, or an eksctl config) provisions an AWS EKS cluster named `kubeflow` with OIDC enabled and the AWS EBS CSI driver for persistent volumes. Juju registers the cluster, bootstraps a controller onto it, and deploys Charmed Kubeflow (pipelines, notebooks, Katib, an Istio ingress, and Dex auth) and Charmed MLflow (tracking, registry, MinIO artifact store). MLflow is integrated into the Kubeflow ingress and dashboard, so the two open-source tools appear as one platform behind a single login.

• ›Terraform modules: VPC and EKS composed in main.tf (bastion and ALB modules exist in the repo but are not wired in)
• ›eksctl ClusterConfig (eks.yaml): EKS `kubeflow`, k8s 1.23, managed Ubuntu node group, OIDC
• ›AWS EBS CSI driver as an EKS addon, permissions via an IRSA role bound through OIDC
• ›Juju operator framework: controller bootstrapped on the EKS cluster
• ›Charmed Kubeflow 1.9: Istio ingress, Dex auth, pipelines, notebooks, Katib
• ›Charmed MLflow 2.15: experiment tracking, model registry, MinIO artifact store
• ›Integration relations that put MLflow behind the Kubeflow ingress and dashboard

Two provisioning paths exist and are kept in sync: reusable Terraform modules and a single eksctl config. The runbook (x.md) uses eksctl.

How one request flows

1. 01Provision the cluster from code: Terraform modules or an eksctl config create an EKS cluster named `kubeflow` with a managed node group and OIDC on.
2. 02Wire storage and identity: install the AWS EBS CSI driver as an addon for persistent volumes, with permissions from a scoped IAM role bound to a service account through OIDC, not from broad node permissions.
3. 03Bring in the operator framework: Juju registers the cluster, bootstraps a controller, and creates a model for the platform.
4. 04Deploy Kubeflow as a bundle: one command deploys pipelines, notebooks, Katib, an Istio ingress, and Dex auth, managed by operators rather than raw manifests.
5. 05Deploy MLflow and connect it: Charmed MLflow with MinIO as its artifact store, then integrated into the Kubeflow ingress and dashboard.
6. 06Hand it over: data scientists log in through the dashboard, work in notebooks, run pipelines, and track runs and models in MLflow. Teardown is a documented pair of commands.

Engineering decisions

What changed along the way

• →Started with a local Kubernetes (microk8s) and Juju to learn the deployment, then moved onto EKS for a real shared cluster.
• →Kept two ways to make the cluster, reusable Terraform modules and an eksctl config, since each suited a different moment.
• →Worked through Juju version churn during setup and settled on pinned charm channels so the deploy is repeatable.

Security and reliability

• ·Scoped identity: OIDC plus IAM roles for service accounts, so components get only the AWS permissions they need.
• ·Private networking: a VPC with public and private subnets, so workloads are not exposed to the internet by default.
• ·Authentication: Dex guards the Kubeflow dashboard behind a login.
• ·Persistent state: the EBS CSI driver backs the platform's volumes and MinIO holds MLflow's artifacts.
• ·Rebuildable: cluster and platform are code, so recovery is a rebuild and teardown is documented.

Metrics

What shipped

• ✓Delivered a working self-hosted MLOps platform: Charmed Kubeflow and Charmed MLflow on AWS EKS, integrated behind one dashboard and login.
• ✓Data scientists get notebooks, pipelines, experiment tracking, and a model registry without leaving the company's AWS account.
• ✓The cluster and platform rebuild from code and pinned versions, and tear down cleanly.
• ✓Persistent storage and artifact storage handled by the EBS CSI driver and MinIO, with scoped IAM throughout.
• ✓A runbook that takes an empty AWS account to a running platform.

Lessons learned

• ·For a platform the size of Kubeflow, an operator framework beats hand-written manifests, because something has to own the lifecycle.
• ·OIDC with per-service-account IAM roles is the right default for cluster permissions, not node-wide grants.
• ·Pinning charm channels is what makes a Kubernetes platform actually reproducible.
• ·Self-hosting the MLOps stack keeps the data in the client's account, which was the whole reason to avoid a SaaS.

Tech stack