Talent.com
Senior Databricks Engineer

Senior Databricks Engineer

Newscape Consultingpune, maharashtra, in
2 days ago
Job description

About the Role

We are seeking a Senior Databricks Engineer to design, develop, and optimize scalable data pipelines and analytics solutions on the Databricks platform. The ideal candidate will be a hands-on expert in data engineering, cloud infrastructure, and distributed data processing, with a deep understanding of how to build reliable and performant data ecosystems to power business insights and AI / ML initiatives.

Key Responsibilities

Design &

  • Development :
  • Build and maintain scalable ETL / ELT data pipelines using Apache Spark and Databricks.
  • Develop Delta Lake architectures and optimize data storage for performance and cost.
  • Automate data ingestion, transformation, and orchestration processes across multiple data sources.
  • Build different stages of medallion architecture such as Bronze, Silver and Gold layer

Data Architecture & Modeling :

  • Collaborate with data architects to design data lakehouse and warehouse solutions.
  • Implement data quality frameworks, versioning, and governance standards (e.g., Unity Catalog).
  • Optimize Spark jobs, cluster configurations, and query performance.
  • Cloud & Infrastructure :
  • Work with Azure Databricks or AWS to configure and manage cloud-based data platforms.
  • Develop and manage CI / CD pipelines for Databricks notebooks, workflows, and jobs.
  • Implement security best practices for data access, encryption, and compliance.

    • Collaboration & Leadership :

  • Partner with data scientists, analysts, and product teams to enable data-driven decisions.
  • Provide technical mentorship to junior engineers.
  • Contribute to strategic planning and architecture reviews.

    • Required Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.
  • 6+ years of experience in data engineering or software development.
  • 4+ years hands-on experience with Databricks and Apache Spark.
  • Strong proficiency in Python, SQL, and PySpark.
  • Experience with Delta Lake, Data Lakehouse, and Data Warehousing concepts.
  • Solid understanding of data modeling, ETL design, medallion architecture and distributed data processing.
  • Familiarity with CI / CD tools (Azure DevOps, GitHub Actions, Jenkins, etc.).
  • Cloud platform expertise in Azure, AWS, or GCP.

    • Preferred Qualifications

  • Experience with machine learning pipelines in Databricks (MLflow).
  • Knowledge of data governance and compliance frameworks.
  • Experience integrating Databricks with BI tools like Power BI, Tableau, or Looker.
  • Certifications such as :
  • Databricks Certified Data Engineer Associate or Professional
  • Azure Data Engineer Associate
  • AWS Big Data Specialty

    • Soft Skills

  • Strong problem-solving and analytical thinking.
  • Excellent communication and collaboration abilities.
  • Proactive mindset with ownership of deliverables.
  • Passion for learning emerging data technologies.

    • Why Join Us

  • Work with cutting-edge data technologies on large-scale projects.
  • Opportunity to shape the organization’s data strategy and architecture.
  • Collaborative and innovative culture focused on continuous growth.
    • Create a job alert for this search

    Senior Engineer • pune, maharashtra, in