Job Title : Azure Data Engineering Lead
Experience : 5+ years
Location : Hyderabad (Hybrid WFO)
Notice Period : Immediate
Role Overview :
We are looking for a highly skilled Azure Data Engineering Lead to design, develop, and optimize scalable data solutions using Azure Databricks and Azure Data Factory (ADF). This role involves collaborating with cross-functional teams to deliver robust data pipelines, enable advanced analytics, and support enterprise data initiatives.
Key Responsibilities :
Data Pipeline Development :
Build and maintain scalable ETL / ELT pipelines using ADF and Databricks notebooks.
Integrate data from diverse sources including Azure Data Lake, SQL DBs, Blob Storage, and external APIs.
Advanced Data Engineering :
Develop distributed data processing workflows using PySpark, Scala, and Delta Lake in Databricks.
Implement data quality checks, error handling, and logging mechanisms.
Automation & DevOps :
Automate deployments using CI / CD pipelines with Azure DevOps and Terraform.
Manage version control and collaborative development using Git.
Performance Optimization :
Monitor and tune pipeline performance for cost-efficiency and scalability.
Troubleshoot and resolve data pipeline failures and bottlenecks.
Collaboration & Documentation :
Work with data scientists, analysts, and business stakeholders to understand data needs.
Document architecture, workflows, and best practices for internal knowledge sharing.
Mandatory Skills : Azure
Azure Databricks
PySpark
Good-to-Have Skills :
Delta Lake
Azure Data Factory / Mapping Data Flows
SQL & Python
Terraform
CI / CD, Git, Agile methodologies
Azure certifications (DP-203, AZ-104)
Data Engineering Lead • Hyderabad, TS, India