Talent.com
This job offer is not available in your country.
Natobotics - Senior Data Engineer - Azure Databricks

Natobotics - Senior Data Engineer - Azure Databricks

Natobotics Technologies Pvt LimitedMumbai
1 day ago
Job description

Description : Senior Data Engineer - Azure ETL / ELT & : Mumbai, MH / Hyderabad, TG / Chennai, TN / Bhubaneswar, OR / Bangalore, : 6-8 Years

About the Role :

We are seeking an experienced Senior Data Engineer with 6-8 years in data engineering to join our team across multiple locations. This role is focused on designing, developing, and maintaining scalable and fault-tolerant data pipelines using the Microsoft Azure ecosystem. The ideal candidate will have deep expertise in Azure Data Factory, Databricks, PySpark, and advanced SQL to drive complex data transformations and implement robust CI / CD :

  • ETL / ELT Pipeline Development : Design, develop, and maintain complex ETL / ELT pipelines for large-scale data ingestion and transformation using Azure Data Factory (ADF) and Azure Databricks.
  • Data Ingestion Flows : Implement robust data ingestion flows to integrate data from diverse sources, including Azure Blob Storage, Azure Data Lake (ADLS), On-Prem SQL, and SFTP, ensuring reliability and security.
  • Data Transformation & Optimization : Design and optimize dimensional and relational data models, and perform complex data transformations utilizing Spark SQL, PySpark, Oracle, SQL Server, and Progress DB SQL.
  • Orchestration Workflow : Build sophisticated orchestration workflows in ADF using a range of activities such as Lookup, ForEach, Execute Pipeline, and Set Variable, to manage complex, multi-stage data jobs.
  • Troubleshooting and Maintenance : Perform root cause analysis and quickly resolve critical production issues affecting data pipelines, Databricks notebooks, and data flow stability.
  • CICD and DevOps : Collaborate with DevOps teams on the creation and enhancement of CI / CD pipelines using tools like Azure DevOps and Jenkins to enable automated testing and deployment of data assets.
  • Performance Tuning : Apply specialized performance tuning techniques to optimize query execution and resource utilization within Azure Synapse Analytics (SQL DW) environments.
  • Documentation and Quality : Maintain comprehensive technical documentation, including detailed runbooks, technical design specifications, and QA test cases, ensuring knowledge transfer and operational efficiency.

Required Skill Set (Mandatory Skills) :

  • Cloud Platforms : 6-8 years of hands-on experience utilizing key Microsoft Azure services for data engineering, including Azure Data Factory (ADF), Azure Databricks (ADB), Azure Data Lake Storage (ADLS), Azure SQL, and Azure Synapse Analytics.
  • Programming & Big Data : Expert proficiency in Python for Data Science, and deep experience with PySpark and Spark SQL for large-scale distributed data processing.
  • ETL Tools : Mandatory expertise in building and maintaining enterprise-level data pipelines using Azure Data Factory.
  • Database & SQL : Expert-level proficiency in SQL for complex querying, performance tuning, and data model design.

    • Preferred Skills :

  • Practical experience applying performance tuning specifically to Azure Synapse Analytics (SQL DW).
  • Experience with Azure DevOps, Git, and CICD pipeline creation using either Azure DevOps or Jenkins.
  • Familiarity with non-relational databases such as Cosmos DB or other NoSQL platforms.
  • Experience ingesting data from older enterprise systems like DB2 or diverse formats like NoSQL sources.

    • (ref : hirist.tech)

    Create a job alert for this search

    Azure Data Engineer • Mumbai