Talent.com
This job offer is not available in your country.
Senior Data Platform Engineer (Azure Databricks)

Senior Data Platform Engineer (Azure Databricks)

Antal InternationalBengaluru, Karnataka, India
1 day ago
Job description

Position Summary :

As the (Senior) Data Platform Engineer (Azure Databricks) , you will be responsible for building, maintaining, and evolving our Azure Databricks platform, ensuring it is secure, reliable, scalable, and cost-efficient. This role is pivotal in empowering our Data- & AIOps team by providing a robust, automated, and well-governed data environment. The ideal candidate will drive a "platform-as-a-product" mindset, continuously improving our architecture, operations, and user experience.

Your Responsibilities :

Platform Architecture & Engineering :

  • Design and implement architectural improvements, including workspace migrations and robust disaster recovery (DR) strategies.
  • Enhance and maintain the platform's security posture, focusing on data governance, data and compute isolation at logical and physical levels.
  • Drive the evolution of the platform by researching, prototyping, and integrating new tools and features to enhance performance and scalability.

Data Governance & Security :

  • Lead the expansion and utilization of Databricks Unity Catalog for granular data governance, including row / column-level security, data lineage tracking, and metadata tagging.
  • Conduct regular security and compliance checks to ensure the platform adheres to internal policies and industry standards.

    • Automation & Operations (DevOps & GitOps) :

  • Build and maintain resilient CI / CD pipelines using GitHub Actions to automate infrastructure (Terraform) and application deployments.
  • Implement and manage a comprehensive observability stack (logging, monitoring, tracing, alerting) to ensure high availability and proactive issue resolution.
  • Apply FinOps principles to monitor, analyze, and optimize platform costs.
  • Manage day-to-day platform administration, including onboarding projects, managing clusters, and configuring jobs.

    • User Enablement & Self-Service :

  • Develop and integrate user-friendly interfaces (e.g., via ServiceNow or Backstage) to enable self-service for access requests and resource provisioning.
  • Create and maintain clear documentation, including architecture diagrams, runbooks, and best-practice guides.
  • Coach and train other colleagues, championing a "platform-as-a-product" mindset focused on delivering value to internal users.

    • Your profile

    Essential Qualifications :

  • Bachelor's or Master's degree in Computer Science, Mathematics, Engineering, or a related discipline.
  • 5+ years of professional experience in a platform engineering, DevOps, or SRE role with a focus on data-intensive platforms.
  • Deep expertise in Microsoft Azure, including ADLSGen2, Private Endpoints, Azure Key Vault, Azure Log Analytics, and EntraID.
  • Extensive hands-on experience with Azure Databricks administration, operations, security, and performance tuning.
  • Strong proficiency with Terraform and a solid understanding of GitOps principles.
  • Advanced skills in shell scripting and experience with a scripting language like Python.
  • Demonstrable experience building CI / CD pipelines with GitHub Actions. Familiarity with Databricks Asset Bundles is a plus.
  • Proficient with Linux administration and troubleshooting.
  • Practical experience implementing data governance solutions for data lakes, especially with Unity Catalog.
  • Solid understanding of Kubernetes (k8s) concepts and architecture.

    • Preferred Qualifications :

  • Understanding of MLOps principles (e.g., feature stores, model deployment strategies) to support ML engineering teams.
  • Experience integrating with or building plugins for platforms like Backstage or ServiceNow.

    • Soft Skills and Cultural Fit :

  • Excellent communication and collaboration skills to work effectively with technical teams and business stakeholders.
  • Strong analytical thinking, problem-solving abilities.
  • Demonstrated high level of initiative, self-motivation, and a proactive, self-starter mindset, with a strong drive to independently identify and solve challenges.
  • A focus on treating the platform as a product, prioritizing the user experience and delivering tangible value to internal customers.
  • A passion for continuous learning, innovation, knowledge sharing, and driving excellence in data engineering.
  • Ability to work effectively in a cross-functional, fast-paced environment.
    • Create a job alert for this search

    Azure Data Engineer • Bengaluru, Karnataka, India