Databricks lead

Designation - Databricks leadExperience - 8+ YearsKey Responsibilities

• Lead the design and implementation ofscalable data pipelines and ETL processes on Databricks.
• Architect and optimize big data and analyticssolutions using Spark, Delta Lake, and MLflow.
• Collaborate with stakeholders to translate businessrequirementsinto technical solutions.
• Oversee the migration of on-premises data systemsto Databricks cloud environments.
• Drive best practicesin data governance, security, and performance optimization.
• Provide technical leadership, mentorship, and support to data engineering teams.
• Work closely with data scientiststo productionize machine learning models.
• Establish monitoring, logging, and automated testing for data pipelines.
• Ensure compliance with industry standards,security protocols, and regulatory requirements.
• Evaluate new featuresin Databricks and recommend adoption strategies.Required Skills & Qualifications
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• 8+ years of experience in data engineering, with 3+ years in Databricks.
• Strong expertise in Apache Spark, PySpark, SQL, and Delta Lake.
• Proven experience with Databricks on Azure, AWS, or GCP.
• Hands-on experience in designing large-scale ETL / ELT workflows.
• Proficiency in cloud storage (S3, ADLS, GCS) and data warehousing solutions (Snowflake, Redshift,Synapse, BigQuery).
• Strong knowledge of DevOps, CI / CD, and Infrastructure as Code (Terraform / CloudFormation).
• Experience in MLflow, feature engineering, and integrating machine learning pipelines.
• Excellent communication, leadership, and stakeholder managementskills.
• Databricks certification(s) such as Databricks Certified Data Engineer Professional or DatabricksCertified Machine Learning Professional preferred.Preferred Skills
• Experience with streaming technologies(Kafka, Event Hubs, Kinesis).
• Knowledge of data governance frameworks(Collibra, Purview).
• Familiarity with BI / visualization tools(Power BI, Tableau, Looker).