Databricks System Engineer

Experience Required : 7+ Years

Work Location : Bangalore / Chennai / Hyderabad

Role Overview :

We are seeking an experienced Databricks Administrator to manage, optimize, and maintain our Databricks platform across multiple environments. The ideal candidate will have a strong background in data engineering infrastructure, cluster management, security, governance, and automation within Databricks and related Azure / AWS ecosystems.

Key Responsibilities :

• Databricks Platform Administration :
• Manage and maintain Databricks workspaces, clusters, jobs, pools, and libraries.
• Monitor performance and optimize resource utilization across environments (dev, test, prod).
• Implement autoscaling, cluster policies, and cost control measures.
• Security & Access Management :
• Configure and maintain user / group access via Azure AD or IAM integration.
• Implement fine-grained access control using Unity Catalog and role-based permissions.
• Ensure compliance with enterprise data security standards.
• Automation & CI / CD :
• Automate cluster provisioning, job deployment, and workspace management using Terraform, Databricks CLI, or REST APIs.
• Integrate Databricks with CI / CD tools like Azure DevOps, Jenkins, or GitHub Actions.
• Monitoring & Troubleshooting :
• Proactively monitor cluster health, job performance, and data pipeline stability.
• Diagnose and resolve system, configuration, and performance issues.
• Implement alerting, logging, and audit trail monitoring.
• Integration & Collaboration :
• Work with Data Engineers, DevOps, and Cloud teams to ensure smooth data operations.
• Support integration with Snowflake, Kafka, Azure Data Lake, or AWS S3.
• Collaborate on optimizing Spark configurations and ETL performance.
• Governance & Compliance :
• Implement best practices for workspace and notebook governance.
• Manage version control for notebooks and data pipelines.
• Enforce tagging, cost allocation, and audit standards.

Required Skills & Qualifications :