Senior Databricks Architect

Skills :

Databricks, Microsoft Azure, Data Engineer, pyspark, kafka, python, Apache Spark, SQL,

About The Role

We are seeking a Senior Databricks Architect to lead the design and implementation of scalable, secure, and cost-optimized data platforms on Azure using Databricks. You will own end-to-end architecture for data ingestion, transformation, governance, and analytics, and mentor engineering teams to deliver high-quality data solutions.

Key Responsibilities

• Architect and implement enterprise-grade data platforms on Azure Databricks (DLT, Workflows, Unity Catalog) with strong security, governance, and reliability.
• Design end-to-end data pipelines : batch and streaming (Structured Streaming), CDC, and near real-time analytics.
• Define and enforce data modeling standards (3NF, dimensional modeling / starsnowflake, data vault where applicable).
• Optimize Databricks workloads (cluster sizing, autoscaling, Delta Lake file management, Z-Ordering, OPTIMIZE / VACUUM) for performance and cost.
• Establish best practices for code (PySpark / Scala / SQL), notebooks, repos / CI-CD, testing, observability, and documentation.
• Integrate with broader Azure ecosystem : ADLS Gen2, ADF / Synapse pipelines, Azure SQL / MI, Azure Key Vault, Event Hubs / Kafka, Azure Functions / Logic Apps.
• Drive data governance, lineage, and access control with Unity Catalog, data masking, and RBAC / ABAC; align with organizational compliance (GDPR / DPDP, SOC2).
• Collaborate with security / network teams on private networking (VNet injection), peering, firewall rules, Private Link, NSGs, VPN / ExpressRoute.
• Lead capacity planning, SLAs / SLOs, incident response, and cost management (photon, serverless, job vs all-purpose clusters).
• Partner with stakeholders (data science, BI, product) to translate business requirements into scalable data solutions.
• Mentor and upskill data engineers; perform code / design reviews; establish architectural guardrails and reference architectures.

Required Qualifications

Preferred / Good-to-Have

Soft Skills

Location

Employment Type

Compensation

How To Apply

Skills Required

Adf, Logic Apps, Pyspark, Apache Spark, Kafka, Sql, Synapse, Terraform, Microsoft Azure, Databricks, Arm, Python