Lead Data Pipeline Architect

Job Summary :

We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview.

Job Description :

• Design, develop, and maintain real-time and batch data pipelines in Databricks.
• Integrate Apache Kafka with Databricks using Structured Streaming.
• Implement robust data ingestion frameworks using Databricks Autoloader.
• Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers.
• Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs.
• Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic.
• Develop reusable components for merge / upsert operations and window function-based transformations.
• Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques.
• Collaborate with cross-functional teams to ensure data availability, reliability, and consistency.

Must Have :

oProcessing Modes : Append, Update, Complete

oOutput Modes : Memory, Console, File, Kafka, Delta

oCheckpointing and fault tolerance

oData partitioning strategies

oCaching and persistence

oAdaptive query execution and cluster configuration tuning

Good to Have :

Please share your CVs shailendra.sharma@celebaltech.com

Immediate Joiners Preferred