Talent.com
Senior Databricks Architect (Mumbai / Hyderabad / Gurgaon)

Senior Databricks Architect (Mumbai / Hyderabad / Gurgaon)

ConfidentialMumbai, India
4 days ago
Job description

Work Locations : Mumbai OR Hyderabad OR Gurgaon

About the Role :

We are seeking a Senior Databricks Architect to lead the design and implementation of scalable, secure, and cost-optimized data platforms on Azure using Databricks. You will own end-to-end architecture for data ingestion, transformation, governance, and analytics, and mentor engineering teams to deliver high-quality data solutions.

Key Responsibilities

  • Architect and implement enterprise-grade data platforms on Azure Databricks (DLT, Workflows, Unity Catalog) with strong security, governance, and reliability.
  • Design end-to-end data pipelines : batch and streaming (Structured Streaming), CDC, and near real-time analytics.
  • Define and enforce data modeling standards (3NF, dimensional modeling / star–snowflake, data vault where applicable).
  • Optimize Databricks workloads (cluster sizing, autoscaling, Delta Lake file management, Z-Ordering, OPTIMIZE / VACUUM) for performance and cost.
  • Establish best practices for code (PySpark / Scala / SQL), notebooks, repos / CI-CD, testing, observability, and documentation.
  • Integrate with broader Azure ecosystem : ADLS Gen2, ADF / Synapse pipelines, Azure SQL / MI, Azure Key Vault, Event Hubs / Kafka, Azure Functions / Logic Apps.
  • Drive data governance, lineage, and access control with Unity Catalog, data masking, and RBAC / ABAC; align with organizational compliance (GDPR / DPDP, SOC2).
  • Collaborate with security / network teams on private networking (VNet injection), peering, firewall rules, Private Link, NSGs, VPN / ExpressRoute.
  • Lead capacity planning, SLAs / SLOs, incident response, and cost management (photon, serverless, job vs all-purpose clusters).
  • Partner with stakeholders (data science, BI, product) to translate business requirements into scalable data solutions.
  • Mentor and upskill data engineers; perform code / design reviews; establish architectural guardrails and reference architectures.

Required Qualifications

  • 8–10 years of professional experience in data engineering / architecture, with at least 3+ years hands-on in Databricks on Azure.
  • Strong data modeling expertise : relational 3NF, dimensional (Kimball), and practical experience with Delta Lake / medallion architecture.
  • Advanced programming skills : PySpark and / or Scala; solid grasp of OOP principles and design patterns in data engineering contexts.
  • Expert SQL skills (analytic / window functions, performance tuning, query optimization).
  • Deep experience with Azure services : ADLS Gen2, ADF / Synapse, Azure SQL / MI, Event Hubs; identity and security (AAD, service principals, managed identities, Key Vault).
  • Hands-on with Databricks features : Delta Live Tables, Unity Catalog, Workflows, Repos / CI-CD, cluster policies, Jobs API.
  • Networking fundamentals for cloud data platforms : VNet design, subnets, routing, Private Link / Endpoints, peering, DNS, firewalling.
  • Database fundamentals : relational and NoSQL (e.g., PostgreSQL, SQL Server, Cosmos DB), indexing, partitioning, transactions.
  • Proven track record building robust ELT / ETL pipelines, CDC (e.g., with Auto Loader, Kafka), and streaming architectures.
  • Experience with DevOps practices : Git, CI-CD (Azure DevOps / GitHub Actions), Infrastructure-as-Code (Terraform / ARM / Bicep preferred).
  • Strong communication, stakeholder management, and leadership / mentoring experience.
  • Bachelor's degree in Computer Science / Engineering or related field from a reputed Indian college / institute.

    • Preferred / Good-to-Have

  • Databricks certifications (Databricks Certified Data Engineer Professional, Machine Learning Professional, or Architect Associate).
  • Experience with BI tools and semantic layers (Power BI, dbt, Tabular models).
  • Exposure to MLOps and feature stores (Databricks Feature Store) and model serving.
  • Cost governance experience : chargeback / showback, cluster policy enforcement, utilization dashboards.
  • Knowledge of data privacy and compliance frameworks (DPDP Act, GDPR, HIPAA as applicable).

    • Skills Required

    Adf, Scala, Pyspark, Kafka, Sql, Synapse, Terraform, Databricks, Arm, Azure

    Create a job alert for this search

    Senior Architect • Mumbai, India