Senior Data Engineer

Description

Responsibilities :

• Develop & Optimize Data Pipelines Build, test, and maintain ETL / ELT data pipelines using Azure Databricks & Apache Spark (PySpark) .Optimize performance and cost-efficiency of Spark jobs.Ensure data quality through validation, monitoring, and alerting mechanisms.Understand cluster types, configuration, and use-case for serverless
• Implement Unity Catalog for Data Governance Design and enforce access control policies using Unity Catalog.Manage data lineage, auditing, and metadata governance .Enable secure data sharing across teams and external stakeholders.
• Integrate with Cloud Data Platforms Work with Azure Data Lake Storage / Azure Blob Storage / Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams .Implement Delta Lake for scalable, ACID-compliant storage.
• Automate & Orchestrate Workflows Develop CI / CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory .Monitor and troubleshoot failures in job execution and cluster performance .
• Collaborate with Stakeholders Work with Data Analysts, Scientists, and Business Teams to understand requirements.Translate business needs into scalable data engineering solutions .
• API expertiseAbility to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience :

Preferred Qualifications :

Soft Skills :