Senior Data Engineer

Job Title : Senior Databricks Engineer

Experience Level : 5-8 Years

Job Summary

As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakehouse Platform. You will leverage your deep expertise in Spark, Delta Lake, and cloud technologies to build scalable and reliable solutions that process large-scale data efficiently. The ideal candidate is a proactive problem-solver with a strong background in data engineering and a passion for mentoring and driving technical excellence.

Key Responsibilities

• Architect and Develop : Design, build, and maintain robust and scalable data pipelines and ETL / ELT processes using Databricks, Apache Spark, and Python / Scala.
• Lakehouse Implementation : Architect and implement solutions on the Databricks Lakehouse Platform, including data modeling, ingestion, storage, and processing using Delta Lake.
• Performance Optimization : Analyze and tune the performance of Spark jobs and Databricks clusters to ensure high efficiency, reliability, and cost-effectiveness.
• Data Governance : Implement and manage data governance, data quality, and security best practices within Databricks, utilizing features like Unity Catalog.
• Collaboration : Work closely with data scientists, analysts, and business stakeholders to understand their data requirements and deliver high-quality, end-to-end data solutions.
• Automation & CI / CD : Develop and maintain CI / CD pipelines for automated testing and deployment of data workflows (e.g., using Azure DevOps, Jenkins, GitHub Actions).
• Mentorship : Mentor junior engineers, conduct code reviews, and champion best practices in data engineering and software development.
• Innovation : Stay current with the latest advancements in the Databricks platform and the broader big data ecosystem, and drive the adoption of new tools and technologies.

Required Qualifications and Skills