Sr. Data Engineer (Databricks)

We are an Australian Data and AI company and are expanding our Databricks Capability in India and are seeking a Senior Databricks Data Engineer to build scalable, production-grade data platforms that enable AI and advanced analytics for our global clients. We are open, transparent, no-nonsense, innovative problem solvers and eternal learners as a team where individual passion and brilliance is combined with a great team spirit to nurture everyone’s journey to be their best and achieve what they aspire.

Role Overview :

As a Senior Databricks Data Engineer , you will architect, design, and develop data pipelines and modeling layers that power analytical and AI-driven workloads. You are likely to work on end-to-end Databricks projects —from ingestion to curation to enablement for ML / AI—while collaborating with consulting and client delivery teams across regions.

Responsibilities :

• Solution Ownership : Take end-to-end responsibility for project delivery, including planning, prioritization, and ensuring data solutions meet quality and performance expectations.
• Stakeholder Collaboration : Engage with business and product teams to understand data needs, translate requirements, and align engineering solutions with strategic goals.
• Cross-Functional Communication : Present complex data concepts in a clear, actionable manner to non-technical stakeholders and leadership.
• Continuous Improvement : Identify process inefficiencies and lead initiatives to improve data workflows, scalability, and team productivity.
• Design and build scalable, reliable data pipelines using Databricks Auto Loader, Delta Live Tables (DLT), and Workflows.
• Develop and maintain curated Medallion architecture layers (Bronze, Silver, Gold) using Delta Lake and Databricks SQL.
• Orchestrate and deploy pipelines using Databricks Asset Bundles (DABs) and integrate with CI / CD tools (GitHub Actions, Azure DevOps).
• Implement data ingestion using Lakeflow Connect and integrate with external sources such as Kafka, ADLS, and REST APIs.
• Develop real-time and batch processing jobs with PySpark, SQL, and Delta Lake.
• Ensure data quality, lineage, and governance using Unity Catalog and DLT Expectations.
• Collaborate with data scientists and ML engineers to prepare feature-ready datasets for MLflow and AI workloads.
• Optimise Databricks clusters and jobs for cost, performance, and reliability.
• Participate in client workshops, provide architectural input, and communicate technical outcomes effectively to business stakeholders.
• Take ownership of projects ensuring exceptional customer outcomes
• Contribute to a data-driven culture that fosters innovation and agility.

Required Skills & Competencies

3 - 5 years of hands-on experience with Databricks , including proficiency in :

Preferred (Nice to have)

What we offer

At our core, we are building a youthful, forward-looking Data & AI brand — where engineers, consultants, and innovators thrive on solving meaningful problems with cutting-edge technologies.

We offer an environment that blends technical depth, consulting exposure, and personal growth in equal measure.