Azure Data Engineer

We are looking for a skilled TL Azure Data Engineer with 9+ years of experience in big data technologies, particularly Python, PYSpark, SQL and data lakehouse architectures. The ideal candidate will have a strong background in building scalable data pipelines and experience with modern data storage formats, including Apache Iceberg. You will work closely with cross-functional teams to design and implement efficient data solutions in a cloud-based environment.

Key Responsibilities :

• Data Pipeline Development :
• Design, build, and optimize scalable data pipelines using Apache Spark.
• Implement and manage large-scale data processing solutions across data lakehouses.
• Data Lakehouse Management :
• Work with modern data lakehouse platforms (e.g.Apache Iceberg) to handle large datasets.
• Optimize data storage, partitioning, and versioning to ensure efficient access and querying.
• SQL & Data Management :
• Write complex SQL queries to extract, manipulate, and transform data.
• Develop performance-optimized queries for analytical and reporting purposes.
• Data Integration :
• Integrate various structured and unstructured data sources into the lakehouse environment.
• Work with stakeholders to define data needs and ensure data is available for downstream consumption.
• Data Governance and Quality :
• Implement data quality checks and ensure the reliability and accuracy of data.
• Contribute to metadata management and data cataloging efforts.
• Performance Tuning :
• Monitor and optimize the performance of Spark jobs, SQL queries, and overall data infrastructure.
• Work with cloud infrastructure teams to optimize costs and scale as needed.

Qualifications :

Nice to Have :