Data Engineer (Databricks)

Job Description :

We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be instrumental in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into data lake and data warehouse. You will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

Key Responsibilities :

1. ETL Development :

• Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
• Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
• Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
• Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
• Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.

2. Data Pipeline Management :

3. Data Analysis and Query Optimization :

4. Project Coordination and Continuous Improvement :

Required Qualifications :

Preferred Qualifications :