Databricks Implementation Lead

About the Role

We are seeking an experienced  Databricks Implementation Lead  to spearhead end-to-end delivery of a large-scale  data engineering and analytics project  built on the  Databricks Lakehouse platform .

The ideal candidate will have deep expertise in  data architecture , hands-on experience with  Databricks (SQL, PySpark, Delta Lake, MLflow) , and the ability to lead teams through  solution design, migration, optimization, and production rollout .

Key Responsibilities

1. Project Leadership

• Own the complete  Databricks implementation lifecycle  from requirement gathering, architecture design, and environment setup to production deployment and handover.
• Act as the primary  technical liaison  between business stakeholders, data engineering teams, and cloud platform teams.
• Ensure alignment with project goals, timelines, and compliance standards ( data governance, security, cost optimization ).

2. Architecture & Design

• Design scalable and secure  data pipelines  leveraging  Databricks, Delta Lake , and  cloud-native services (Azure, AWS, or GCP) .
• Define the  data ingestion, transformation, and consumption architecture , integrating multiple data sources (APIs, databases, files, streaming).
• Establish  CI / CD pipelines  for Databricks notebooks and jobs, enabling automation and version control (using Git / Azure DevOps).

3. Technical Implementation

• Lead and mentor a team of  data engineers  to develop  ETL / ELT pipelines  using  PySpark, SQL, and Databricks workflows .
• Configure clusters, jobs, and  Lakehouse storage  following best practices.
• Optimize  Databricks compute and query performance  for cost and efficiency.
• Implement  data quality frameworks, monitoring dashboards, and error-handling logic .

4. Governance & Operations

• Set up  access control, role-based security, and data lineage tracking  in  Unity Catalog .
• Collaborate with the  CloudOps team  for environment setup, monitoring, and cost governance.
• Drive  documentation, training, and knowledge transfer  to ensure sustainability and handover readiness.

Required Skills & Experience

• 8+ years of experience in  data engineering or big data projects .
• 3+ years of  hands-on Databricks experience  (SQL, PySpark, Delta Lake, MLflow).
• Strong experience with  ETL / ELT pipeline design ,  data modeling , and  data warehousing .
• Experience with  Azure Data Factory, AWS Glue , or similar orchestration tools.
• Proficiency in  Python, SQL , and  Spark optimization techniques .
• Experience implementing  DevOps practices (CI / CD, Git, Databricks Repos) .
• Familiarity with  data governance frameworks, Unity Catalog, and RBAC .
• Proven track record in  leading multi-member implementation teams .

Preferred Qualifications

• Databricks Certified Data Engineer Professional  or  Solution Architect certification .
• Experience with  AI / ML workflows  and model management using  MLflow .
• Exposure to  Power BI, Tableau, or Looker  for downstream visualization.
• Strong communication skills and experience in  client-facing roles .

Soft Skills

• Excellent  problem-solving and analytical skills .
• Strong  leadership and mentoring  capabilities.
• Ability to work in a  fast-paced, cross-functional environment .
• Clear  documentation and presentation  skills.

Skills Required

Pyspark, AWS Glue, Sql, ELT, Git, Azure Data Factory, rbac, Databricks, Python, Etl