Key Responsibilities:
- Architect and implement enterprise-grade Lakehouse solutions using Databricks.
- Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL).
- Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks.
- Implement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows.
- Design large-scale data warehousing solutions with 3NF and dimensional modeling.
- Establish data governance, security, and data quality frameworks, including Unity Catalog.
- Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI).
- Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow).
- Drive CI/CD, DevOps practices, and performance optimization of Spark workloads.
- Provide technical leadership, mentor teams, and ensure successful delivery.
- Collaborate with stakeholders to translate business requirements into scalable solutions.
Ideal Candidate:
- Databricks Lead.
- 10+ years of total experience, with 5+ years specifically in Databricks.
- Architecture-level expertise must understand end-to-end design, not just engineering execution.
- Hands-on with Delta Lake, Delta Live Tables, Spark (PySpark/SQL), and Azure Data Factory.
- Prior experience working on Azure or Azure DevOps environments.
- Has handled US/UK or international client-facing projects.
- Comfortable managing and leading a team.
- Basic understanding of AI/ML tools and services is a plus.
Perks, Benefits & Culture:
- The company provides free AWS and Azure certification training.
- Group medical insurance of ?5 lakhs is included in the benefits package, along with meal allowances.
Data Engineer - Databricks/Python • Chennai
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for developing, optimize, and maintaining business intelligence and data warehouse systems, ensuring secure, efficient data storage and retrieval, enabling self-service data exploration... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
Responsible for designing and implementing data pipelines, ensuring data quality and accessibility, creating data models, managing ETL processes, optimizing databases, collaborating with stakeholde... Show more
