Data Engineering Azure databricks

Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.

Responsibilities :

• Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
• Create the data integration and data diagram documentation.
• Lead the data validation, UAT and regression test for new data asset creation.
• Create and maintain data models, including schema design and optimization.
• Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills :

Skills Required

Pyspark, Hive, Hadoop, Hue, Apache Oozie, Databricks, Greenplum, Azure, Sql, Python