Senior Data Engineer - Hive / Spark

Description : Role Summary :

We are seeking Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kubernetes-based data platform. The role requires strong hands-on development skills in data engineering, with the ability to deliver high-quality pipelines under guidance from internal leads.

Key Responsibilities :

• Develop and optimize data pipelines using Spark 3.5 and Python / Scala.
• Migrate existing Hive, Spark, and Control-M jobs to Airflow and DBT-based workflows.
• Integrate data pipelines with messaging systems (Kafka, Solace) and object stores (S3, MinIO).
• Troubleshoot and optimize distributed jobs running in Kubernetes environments.
• Collaborate closely with internal leads and architects to implement best practices.
• Design and implement migration / acceleration framework to automate end to end migration.
• Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.
• Work with various data applications to enable and support the migration process.
• Deliver assigned migration tasks within agreed timelines.

Required Skills :

(ref : hirist.tech)