Azure Data Engineering Lead

Job Title : Azure Data Engineering Lead

Experience : 5+ years

Location : Hyderabad (Hybrid WFO)

Notice Period : Immediate

Role Overview :

We are looking for a highly skilled Azure Data Engineering Lead to design, develop, and optimize scalable data solutions using Azure Databricks and Azure Data Factory (ADF). This role involves collaborating with cross-functional teams to deliver robust data pipelines, enable advanced analytics, and support enterprise data initiatives.

Key Responsibilities :

Data Pipeline Development :

Build and maintain scalable ETL / ELT pipelines using ADF and Databricks notebooks.

Integrate data from diverse sources including Azure Data Lake, SQL DBs, Blob Storage, and external APIs.

Advanced Data Engineering :

Develop distributed data processing workflows using PySpark, Scala, and Delta Lake in Databricks.

Implement data quality checks, error handling, and logging mechanisms.

Automation & DevOps :

Automate deployments using CI / CD pipelines with Azure DevOps and Terraform.

Manage version control and collaborative development using Git.

Performance Optimization :

Monitor and tune pipeline performance for cost-efficiency and scalability.

Troubleshoot and resolve data pipeline failures and bottlenecks.

Collaboration & Documentation :

Work with data scientists, analysts, and business stakeholders to understand data needs.

Document architecture, workflows, and best practices for internal knowledge sharing.

Mandatory Skills : Azure

Azure Databricks

PySpark

Good-to-Have Skills :

Delta Lake

Azure Data Factory / Mapping Data Flows

SQL & Python

Terraform

CI / CD, Git, Agile methodologies

Azure certifications (DP-203, AZ-104)