Migration from Apache Cassandra on EC2 to Apache Cassandra on AWS EKS
Migrating a Live Cassandra Datastore Between Clusters
A consumer marketplace kept its data in Apache Cassandra and needed to move it from a cluster running on EC2 to a new cluster running on Kubernetes. I planned and ran the migration with DataStax's Cassandra Data Migrator on Spark, table by table, preserving each row's time-to-live and confirming the counts matched on both sides. The hard part was not the copy, it was the schema: complex user-defined types, collections, and an event-sourced journal that all had to survive.
The problem
A live Cassandra database on EC2 had to become a Cassandra database on AWS EKS, without losing a row. Cassandra holds a lot of data and is always being written to, so a naive dump-and-load risks missing recent writes, resetting expiry timers, or quietly dropping rows the tool cannot parse. This data was not simple: one table nested user-defined types and lists inside each row, another was an event journal from an event-sourced service where the order of events and the raw blobs matter, and many rows carried a time-to-live that had to be kept rather than reset. The move needed a tool built for Cassandra, a plan for the awkward schemas, and a way to prove afterwards that nothing was lost.
Constraints
- ·Live production data, so the migration had to be safe against a working cluster and repeatable.
- ·Preserve time-to-live, so rows with an expiry keep their remaining lifespan rather than restarting the clock.
- ·Complex schema: frozen user-defined types, collections, and an event-sourced journal all had to come across.
- ·Prove no data loss: the result had to be verifiable, not assumed.
Architecture
The DataStax Cassandra Data Migrator runs as a Spark job (spark-submit, local mode) that reads each keyspace table from the source Cassandra cluster on EC2 and writes it to the target Cassandra cluster on AWS EKS, preserving time-to-live. The table with nested user-defined types and collections is written into a flattened, no-collections target table. After each table, a COUNT(*) on the source and target confirms row parity.
- ›Source: Apache Cassandra on EC2 (tables dog, walkpool, messages)
- ›Migration engine: DataStax Cassandra Data Migrator (cassandra-data-migrator-4.1.16-SNAPSHOT), class com.datastax.cdm.job.Migrate
- ›Runner: Apache Spark 3.5.1 (spark-submit, --master local[*], 25G driver and executor), config in cdm.properties
- ›Target: Apache Cassandra on AWS EKS (walkpool migrated into walkpool_nocollections)
- ›Validation: SELECT COUNT(*) on source vs target, per table
- ›Access: cqlsh with source/target credentials (in .env)
How one request flows
- 01Point at both clusters: the migrator connects to the source and target with their own credentials, defined in a properties file, one table at a time.
- 02Run the job on Spark: a spark-submit runs the migrator's Migrate job for one keyspace and table, reading rows in parallel and writing them to the target.
- 03Keep the expiry: for rows with a time-to-live, the migrator carries the remaining lifespan across, so the row's clock is not reset.
- 04Handle the awkward tables: the table with nested types and collections is written into a flattened, no-collections target table.
- 05Prove it landed: a row count on the source and target is compared for that table, and equal counts mean the table came across whole.
Engineering decisions
A tool built for Cassandra, not a hand-rolled script
What. Used DataStax Cassandra Data Migrator on Spark rather than a custom copy script.
Why. It reads and writes in parallel and understands Cassandra's tokens and time-to-live, which a hand-rolled script would have to reinvent.
Tradeoff. A pre-release build with thin documentation, so the configuration had to be worked out by hand, still better than reinventing partitioning and TTL.
The complex-type table
What. The table nesting user-defined types and lists was migrated into a flattened, no-collections target table.
Why. The migrator could not carry the frozen UDT and collection columns across as they were.
Tradeoff. A denormalized target for that one table, taken deliberately so the migration could complete rather than fail on a column type.
Migrating an event journal
What. The event-sourced journal (messages) was migrated as-is, keeping sequence order and binary payloads.
Why. Its order and blobs are the service's history and cannot be reshaped safely.
Tradeoff. No transformation possible, so it moves verbatim.
Proving nothing was lost
What. Compared row counts on source and target after each table.
Why. Zero data loss is a claim that needs evidence, and a count is a simple honest check.
Tradeoff. A count confirms completeness, not per-row equality, which was acceptable here.
What changed along the way
- →The migrator's documentation was thin, so much of the work was reading its behavior and settling on a configuration that preserved time-to-live correctly.
- →The nested-type table forced a schema decision mid-way: rather than fight the tool, the target became a flattened, no-collections table.
- →Each table was migrated and validated on its own, so a problem with one never risked the others.
Security and reliability
- ·Credentials: source and target Cassandra credentials are kept out of the code, in an environment file.
- ·Non-destructive: the migration reads from the source and writes to a fresh target, so the original cluster is never altered.
- ·Repeatable: each table is a separate job that can be re-run, with the count check confirming the result every time.
- ·Expiry preserved: time-to-live carries across, so data meant to expire still expires on schedule.
Metrics
What shipped
- ✓Migrated the tables from the EC2 cluster to the EKS cluster with row counts matched on both sides.
- ✓Preserved time-to-live, so expiring data still expires on schedule.
- ✓Handled the nested-type table with a flattened target, and moved the event journal with its order and blobs intact.
- ✓Captured the migration as a repeatable runbook, so any table can be re-run and re-validated.
Lessons learned
- ·For a real Cassandra migration, a purpose-built tool on Spark beats a hand-rolled script, because it already knows tokens and time-to-live.
- ·Complex column types force a schema decision, and a flattened target is a fair trade to let the move complete.
- ·An event-sourced journal is migrated, not reshaped, so its history stays correct.
- ·A row count on both sides is the simplest honest proof that nothing was lost.