Lead Data Pipeline Architect

Tech Stack :

Python

Kafka

Apache Flink / Apache Spark (Streaming)

Apache Hudi / Apache Iceberg

Required skills :

• 5+ years of experience as a Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines.
• 3+ years of experience designing and running real-time data streaming systems (Kafka + Flink / Spark Streaming).
• 3+ years of proficiency in Python for data engineering (data processing, orchestration, automation).
• Solid understanding of distributed systems, data partitioning, checkpointing, and fault-tolerant stream processing.
• Practical experience with Apache Hudi or Apache Iceberg for incremental data storage and schema evolution.
• Experience with RDBMS sources (PostgreSQL, MySQL, etc.) and data lakes / object storage (S3, GCS, etc.).
• Deep understanding of ETL / ELT design patterns, data modeling, and data quality principles.
• Experience deploying and maintaining data pipelines in AWS is preferred.
• Excellent analytical and problem-solving skills, with the ability to design robust, scalable, and efficient architectures.
• Strong communication skills and ability to collaborate with cross-functional teams.