Data Engineer - ETL / PySpark

We are building a next-generation Customer Data Platform (CDP) powered by the Databricks Lakehouse architecture and Lakehouse Engine framework. We're looking for a skilled Data Engineer with 4-9 years of experience to help us build metadata-driven pipelines, enable real-time data processing, and support marketing campaign orchestration capabilities at scale.

Responsibilities :

• Configure and extend the Lakehouse Engine framework for batch and streaming pipelines.
• Implement the medallion architecture (Bronze -> Silver -> Gold) using Delta Lake.
• Develop metadata-driven ingestion patterns from various customer data sources.
• Build reusable transformers for PII handling, data standardization, and data quality enforcement.
• Build Spark Structured Streaming pipelines for customer behavior and event tracking.
• Set up Debezium + Kafka for Change Data Capture (CDC) from CRM systems.
• Design and develop identity resolution logic across both streaming and batch datasets.
• Use Unity Catalog for managing RBAC, data lineage, and auditability.
• Integrate Great Expectations or similar tools for continuous data quality monitoring.
• Set up CI / CD pipelines for deploying Databricks notebooks, jobs, and DLT pipelines.

Requirements :

(ref : hirist.tech)