Senior Data Engineer

Description

:

Responsibilities:

• Develop & Optimize Data PipelinesBuild, test, and maintainETL/ELTdata pipelines using AzureDatabricks & Apache Spark (PySpark).Optimizeperformance and cost-efficiencyof Spark jobs.Ensure data quality through validation, monitoring, and alerting mechanisms.Understand cluster types, configuration, and use-case for serverless
• Implement Unity Catalog for Data GovernanceDesign and enforceaccess control policiesusing Unity Catalog.Managedata lineage, auditing, and metadata governance.Enable secure data sharing across teams and external stakeholders.
• Integrate with Cloud Data PlatformsWork withAzure Data Lake Storage / Azure Blob Storage/ Azure Event Hubto integrate Databricks with cloud-baseddata lakes, data warehouses, and event streams.ImplementDelta Lakefor scalable, ACID-compliant storage.
• Automate & Orchestrate WorkflowsDevelopCI/CDpipelines for data workflows usingAzureDatabricks Workflows or Azure Data Factory.Monitor and troubleshoot failures injob execution and cluster performance.
• Collaborate with StakeholdersWork withData Analysts, Scientists, and Business Teamsto understand requirements.Translate business needs intoscalable data engineering solutions.
• API expertiseAbility to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:

• Azure Databricks & Apache Spark (PySpark)– Strong experience in buildingdistributed data pipelines.
• Python– Proficiency in writing optimized and maintainable Python code for data engineering.
• Unity Catalog– Hands-on experience implementingdata governance, access controls, and lineage tracking.
• SQL– Strong knowledge of SQL for data transformations and optimizations.
• Delta Lake– Understanding oftime travel, schema evolution, and performance tuning.
• Workflow Orchestration– Experience withAzureDatabricks Jobs or Azure Data Factory.
• CI/CD & Infrastructure as Code (IaC)– Familiarity with Databricks CLI, Databricks DABs, and DevOps principles.
• Security & Compliance– Knowledge ofIAM, role-based access control (RBAC), and encryption.

Preferred Qualifications:

• Experience withMLflowfor model tracking & deployment in Databricks.
• Familiarity withstreaming technologies(Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
• Hands-on experience withdbt (Data Build Tool)for modular ETL development.
• Certification inDatabricks, Azure is a plus.
• Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
• Experience with Azure Synapse Link for Dynamics, dataverse
• Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc

Soft Skills:

• Strongproblem-solvingand debugging skills.
• Ability towork independently and in teams.
• Excellentcommunication and documentationskills.