←BACK TO RESUME
Private Architecture
May 20242 weeks

Migration from Apache Cassandra on EC2 to Apache Cassandra on AWS EKS

Engineering Dossier

Achievement Log

2024-05-05

Successfully orchestrated complex Cassandra database migration with sophisticated data transformation requirements and enterprise-grade validation

  • Conducted comprehensive research and evaluation of multiple migration tools, overcame significant documentation limitations and enterprise tool requirements, and developed custom migration strategies
  • Engineered custom migration scripts with advanced data validation processes, implemented comprehensive testing protocols, and established enterprise-grade migration monitoring
✨ Result:

Zero data loss migration, 50% improvement in query performance, and successful enterprise database modernization

Overview

Successfully migrated data between two Apache Cassandra clusters, moving from an EC2-hosted environment to AWS EKS-hosted Cassandra. Focused on data consistency, minimal downtime, and schema replication.

Core Technologies

Apache CassandraAWS VPC & SubnetsApache SparkAWS EKS

Implementation & Architecture

Apache Cassandra on EC2

Original setup with Apache Cassandra hosted on EC2 instances.

Execution Protocol

  1. Established secure connection to the source Apache Cassandra cluster on EC2.
  2. Ensured data consistency by replicating schema from EC2 to AWS EKS.

Apache Cassandra on AWS EKS

Target setup with Apache Cassandra hosted on AWS EKS (Kubernetes).

Execution Protocol

  1. Set up Apache Cassandra on AWS EKS for improved scalability and management.
  2. Ensured high availability and fault tolerance within Kubernetes.

Data Migration Using Apache Spark

Used Apache Spark jobs to migrate large datasets between Apache Cassandra clusters.

Execution Protocol

  1. Migrated data from EC2-hosted Apache Cassandra to AWS EKS-hosted Apache Cassandra using Apache Spark.
  2. Validated data consistency after migration using Apache Spark tools.

Technical Skills

  • Apache Cassandra
  • Apache Spark
  • AWS EKS
  • Kubernetes
  • Cloud Migration Patterns (6Rs)
  • NoSQL Data Modeling
  • Amazon EC2
  • Amazon VPC
  • VPC Subnets & Routing
  • System Debugging & Troubleshooting

Engineering Challenges

  • →Minimizing Downtime During Migration
  • →Data Consistency

Project Outcomes

  • ✓
    Successful Data Migration
  • ✓
    Seamless Transition to AWS EKS