Azure Data Engineer

Job Description : Responsibilities :

• Design, develop, and maintain scalable and reliable data pipelines using Azure Data Factory (ADF) or Azure Synapse Pipelines to ingest, process, and transform large volumes of data from various sources (on premises, cloud, APIs, etc.).
• Build and optimize data warehousing solutions on Azure Synapse Analytics (dedicated SQL pools or serverless SQL pools) or Azure SQL Database, ensuring efficient data storage, retrieval, and performance.
• Implement and manage data lakes on Azure Data Lake Storage (ADLS Gen2) for storing structured, semi-structured, and unstructured data.
• Develop and implement data quality frameworks and processes to ensure data accuracy, consistency, and integrity throughout the data lifecycle.
• Design and implement data governance policies and procedures in collaboration with data governance teams.
• Utilize Azure Databricks with Spark (PySpark or Scala) for advanced data processing, transformations, and analytics.
• Implement data streaming solutions using Azure Event Hubs, Azure IoT Hub, or Azure Stream Analytics for real-time data ingestion and processing.
• Develop and deploy data integration solutions using various Azure services and custom code as needed.
• Monitor and troubleshoot data pipelines, data warehousing solutions, and data quality processes, ensuring high availability and performance.
• Implement security measures for data storage and processing in Azure, adhering to best practices and compliance requirements.
• Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and deliver solutions that meet their needs.
• Automate data engineering tasks and deployments using Infrastructure-as-Code (IaC) tools like ARM templates or Terraform.
• Stay up-to-date with the latest Azure data services and technologies and evaluate their potential application to our data architecture.
• Optimize data pipelines and queries for performance and cost efficiency.
• Document data pipelines, data models, and data engineering processes.
• Participate in code reviews and ensure adherence to coding standards and best practices.
• Potentially mentor and guide junior data engineers within the team.

Technical Qualifications :

1. Azure Data Factory (ADF) or Azure Synapse Pipelines : Strong proficiency in creating and managing complex ETL / ELT pipelines.

2. Azure Synapse Analytics (Dedicated SQL Pools and / or Serverless SQL Pools) or Azure SQL Database : Deep understanding of data warehousing concepts, schema design, performance tuning, and SQL.

3. Azure Data Lake Storage (ADLS Gen2) : Experience in designing and managing data lakes.

4. Azure Databricks : Strong experience with Spark (PySpark or Scala) for large-scale data processing and analytics.

5. Azure Blob Storage : Understanding of different storage tiers and best practices.

Preferred Qualifications :

ref : hirist.tech)