Experience Required : 7+ Years
Work Location : Bangalore / Chennai / Hyderabad
Role Overview :
We are seeking an experienced Databricks Administrator to manage, optimize, and maintain our Databricks platform across multiple environments. The ideal candidate will have a strong background in data engineering infrastructure, cluster management, security, governance, and automation within Databricks and related Azure / AWS ecosystems.
Key Responsibilities :
- Databricks Platform Administration :
- Manage and maintain Databricks workspaces, clusters, jobs, pools, and libraries.
- Monitor performance and optimize resource utilization across environments (dev, test, prod).
- Implement autoscaling, cluster policies, and cost control measures.
- Security & Access Management :
- Configure and maintain user / group access via Azure AD or IAM integration.
- Implement fine-grained access control using Unity Catalog and role-based permissions.
- Ensure compliance with enterprise data security standards.
- Automation & CI / CD :
- Automate cluster provisioning, job deployment, and workspace management using Terraform, Databricks CLI, or REST APIs.
- Integrate Databricks with CI / CD tools like Azure DevOps, Jenkins, or GitHub Actions.
- Monitoring & Troubleshooting :
- Proactively monitor cluster health, job performance, and data pipeline stability.
- Diagnose and resolve system, configuration, and performance issues.
- Implement alerting, logging, and audit trail monitoring.
- Integration & Collaboration :
- Work with Data Engineers, DevOps, and Cloud teams to ensure smooth data operations.
- Support integration with Snowflake, Kafka, Azure Data Lake, or AWS S3.
- Collaborate on optimizing Spark configurations and ETL performance.
- Governance & Compliance :
- Implement best practices for workspace and notebook governance.
- Manage version control for notebooks and data pipelines.
- Enforce tagging, cost allocation, and audit standards.
Required Skills & Qualifications :
Bachelor’s degree in Computer Science, Engineering, or related field.7+ years of experience in Data Engineering or Platform Administration, with at least 3+ years in Databricks Administration .Strong understanding of Apache Spark architecture and performance tuning.Hands-on experience with Azure Databricks or AWS Databricks environments.Proficiency with Terraform , Databricks CLI , REST APIs , and CI / CD pipelines .Expertise in cluster configuration, job scheduling, and resource optimization .Familiarity with Unity Catalog , Delta Lake , DBFS , and Lakehouse architecture .Experience integrating Databricks with ADLS, S3, Snowflake, Synapse, or Redshift .Strong problem-solving and scripting skills (Python, Shell, or PowerShell).