Azure Data Engineer

Position : Data Engineer

Department : Data & AI

ROLE OVERVIEW

Responsible for designing and maintaining scalable data pipelines on Microsoft Fabric and Azure. Focus includes ingesting structured, semi-structured, and unstructured data, managing OneLake / Delta Lake, and implementing CI / CD-driven deployments for analytics and AI workloads.

KEY RESPONSIBILITIES

• Design and build end-to-end pipelines across Bronze, Silver, and Gold layers in Microsoft Fabric.
• Ingest and transform structured (SQL), semi-structured (JSON, Parquet, CSV), and unstructured data (PDFs, logs, images).
• Develop Fabric Dataflows Gen2, Spark notebooks, and Data Pipelines for batch or near-real-time ingestion.
• Manage OneLake and Delta Lake tables (partitioning, schema evolution, performance tuning).
• Implement data quality checks, metadata tagging, lineage, and logging processes.
• Integrate curated data with Power BI semantic models and downstream AI workloads.
• Apply governance using Microsoft Purview, including sensitivity labels and lineage tracking.
• Configure secure data access using RLS, ACLs, and Azure Key Vault integration.
• Implement CI / CD pipelines for Fabric, ADF, Synapse, or Databricks workloads using Azure DevOps or GitHub Actions.
• Manage version control, branching strategies, automated testing, and deployment workflows.
• Monitor data pipelines using Monitoring Hub, Azure Monitor, and Log Analytics.

QUALIFICATIONS