Senior Deltalake Engineer

Extensive hands-on experience in Partitioning Delta Lake Tables to achieve optimized query performance and storage efficiency.

Proficient in reading, interpreting, and reporting Delta Lake Transaction Logs to ensure data reliability and consistency.

Strong expertise in transforming streaming data and building Delta Live Tables (DLT) pipelines for real-time analytics.

Proven ability to design and implement Slowly Changing Dimensions (SCDs) using PySpark for scalable data warehousing solutions.

Skilled in configuring and leveraging Auto Optimize to enhance Delta Lake performance.

Practical knowledge of Directed Acyclic Graphs (DAGs) for workflow orchestration (e.g., Airflow, Databricks Workflows).

Proficient in developing and managing Materialized Views for accelerating queries and optimizing performance.

In-depth understanding of Medallion Architecture (Bronze, Silver, Gold layers) to structure data pipelines in a Lakehouse environment.

Strong focus on performance tuning and PySpark best practices, including :

Efficient implementation of Window Functions

Optimized Merge operations for handling large-scale datasets