No longer accepting applications
Lead Databricks Platform Engineer
SonatypeHyderabad, Republic Of India, IN3 hours ago
Job descriptionUser Provisioning and Management : Onboard and offboard users, groups, and service principals within Databricks, including integration with identity providers (IdPs) like Azure Active Directory or Okta via SCIM. Manage user roles and entitlements at both the account and workspace levels (Account Admins, Workspace Admins, Metastore Admins, etc.). Implement and maintain role-based access control (RBAC) and attribute-based access control (ABAC) to ensure appropriate data and resource access. Data Lake Governance (Unity Catalog focus) : Configure and manage Unity Catalog metastores, catalogs, schemas, and tables. Define and enforce data access policies (e.G., table-level, column-level, row-level security) using Unity Catalog. Manage data lineage and auditing capabilities to track data flow and usage. Collaborate with data owners and stakeholders to define data quality standards and ensure data integrity.Implement data retention and lifecycle management policies. Aligning Data Sensitivity Policy to Enforceable Data Governance : Translate organizational data classification and sensitivity policies into technical controls within Databricks. Utilize features like data masking and encryption to protect sensitive information. Ensure compliance with regulatory requirements (e.G., GDPR, HIPAA, CCPA) by implementing appropriate security measures. Conduct regular security audits and vulnerability assessments. Managing Cluster and Budget Policies : Define and implement compute policies to control cluster creation, configuration, and resource usage, ensuring cost optimization. Monitor and manage serverless budget policies to attribute usage to specific teams or projects. Optimize cluster configurations for performance and cost-effectiveness, leveraging features like auto-scaling and auto-termination. Manage cluster pools to reduce startup times and improve resource allocation. Reporting and Addressing Discrepancies : Monitor Databricks platform health, performance, and resource utilization. Identify and troubleshoot issues related to user access, data availability, cluster performance, and policy violations. Generate reports on platform usage, costs, security incidents, and compliance. Investigate and resolve discrepancies in data, reports, or system behavior in collaboration with data engineers, data scientists, and other teams. Develop and maintain comprehensive documentation of configurations, procedures, and best practices. Collaboration and Support : Provide technical support and guidance to Databricks users, data engineers, and data scientists. Collaborate with cloud infrastructure teams (AWS, Azure, GCP) to manage underlying cloud resources. Stay up-to-date with the latest Databricks features, best practices, and industry trends. Databricks Platform Expertise : Deep understanding of Databricks architecture, workspaces, and key components (Unity Catalog, Delta Lake, Spark, SQL Analytics).Proficiency in Databricks administration console and APIs. Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT) for orchestration and pipeline management. Cloud Platform Knowledge : Strong experience with AWS and its relevant services. Data Governance & Security : Solid understanding of data governance principles, data classification, and data lifecycle management. Experience implementing security controls, access policies (RBAC), and encryption. Familiarity with compliance standards (GDPR, HIPAA, CCPA) and auditing practices. Programming & Scripting : Proficiency in SQL for data querying and access control. Deep expertise in Terraform is essential, extending beyond basic knowledge to managing complex, multi-project infrastructure. This includes hands-on experience with custom Terraform modules crucial for Data Mesh orchestration. Scripting skills (e.G., Python, Terraform) for automation and administrative tasks. Familiarity with Spark and PySpark concepts for troubleshooting and optimization. Identity and Access Management (IAM) : Experience with enterprise identity providers (e.G., Azure AD, Okta, Active Directory) and SCIM provisioning. Networking Concepts : Understanding of network security, VPNs, VPCS, private links, VPC peering, and connectivity within cloud environments. Monitoring & Logging Tools : Experience with monitoring tools (e.G., Datadog, Observe, cloud-native monitoring) for platform health and performance. Problem-Solving and Troubleshooting : Ability to diagnose and resolve complex technical issues efficiently. Communication : Excellent verbal and written communication skills to interact with technical and non-technical stakeholders. Attention to Detail : Meticulous in configuring policies, managing access, and ensuring data integrity. Proactive and Self-Driven : Ability to anticipate issues, recommend solutions, and continuously improve the platform. Collaboration : Work effectively with cross-functional teams (data engineers, data scientists, security teams). Analytical Thinking : Ability to analyze data and system logs to identify trends and discrepancies.
Role Summary
- The Databricks Administrator will be responsible for the overall health, security, and performance of the Databricks platform. This includes managing user access, implementing and enforcing data governance policies, optimizing cluster resources, and ensuring data sensitivity policies are effectively applied across the data lakehouse. The administrator will also be crucial in identifying, reporting, and resolving discrepancies within the platform's operation and configuration.
Key Responsibilities
Technical Skills :
Soft Skills
Create a job alert for this search
Platform Engineer • Hyderabad, Republic Of India, IN
Related jobs
- Promoted
Data Engineer – Databricks Platform
Amicon Hub ServicesHyderabad, Telangana, India Delta Lake, Spark, PySpark, SQL).SQL Server, MongoDB, InfluxDB).Kafka, Azure Event Hubs, or similar).Excellent problem-solving skills and the ability to work in a fast-paced environment.Familiar wi...Show moreLast updated: 28 days ago
- Promoted
Data Engineer Lead
JRD SystemsHyderabad, Telangana, India About the Role We are seeking an experienced Data Engineer Lead to design, develop, and maintain scalable data solutions on Azure and Databricks as part of our enterprise data modernization initia...Show moreLast updated: 3 days ago
- Promoted
Data Engineer
IntraEdgeHyderabad, IN We are seeking a highly skilled Data Engineer with strong experience in Python, PySpark, Snowflake, and AWS Glue to join our growing data team.
You will be responsible for building scalable and reli...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer - Snowflake
Prudent Technologies and Consulting, Inc.hyderabad, telangana, in We are seeking a skilled Data Engineer with strong experience in Python, Snowflake, and AWS.The ideal candidate will be responsible for building and optimizing scalable data pipelines, integrating ...Show moreLast updated: 4 days ago
- Promoted
AWS Data Engineer
Tata Consultancy Servicessecunderabad, telangana, in TCS is Hiring AWS Data Engineer Bangalore location.Strong hands-on experience in Python programming and PySpark.Experience using AWS services (RedShift, Glue, EMR, S3 & Lambda).Experience working w...Show moreLast updated: 26 days ago
- Promoted
Senior DataOps Engineer (AWS)
MSBC Grouphyderabad, India Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services.Lead the E-Comms data pipeline within Compass’s Application Simplification workstream : design, ...Show moreLast updated: 23 hours ago
- Promoted
Lead Data Engineer - Python / AWS
Zorba Consulting India Pvt. Ltd.Hyderabad Description : Primary Job Title : Data Engineering Lead.About The Opportunity : We are seeking a highly skilled Lead Data Engineer with...Show moreLast updated: 26 days ago
- Promoted
Databricks Engineer
TTC GroupHyderabad, IN We are seeking a Mid-Level Databricks Engineer with strong data engineering fundamentals and hands-on experience building scalable data pipelines on the Databricks platform.The ideal candidate will...Show moreLast updated: 1 day ago
- Promoted
Databricks Data Engineer Lead – Sustainability Project
Blue Cloud Softech Solutions LimitedHyderabad, IN BCSS is seeking a Databricks Data Engineer to support its enterprise-wide Sustainability initiative.The engineer will be responsible for building data pipelines and models to support product-level ...Show moreLast updated: 3 days ago
- Promoted
- New!
Lead Data Solutions Engineer
KPI PartnersHyderabad, Republic Of India, IN We are seeking a highly skilled Principal Data Engineer to join our dynamic team at KPI Partners.Lead data engineering initiatives and projects from conception to delivery.Design and implement scal...Show moreLast updated: 3 hours ago
- Promoted
Egen - Lead Data Engineer - Google Cloud Platform
SPRINGML INDIA DEVELOPMENT CENTER PRIVATE LIMITEDHyderabad Job Overview : We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engi...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer (Onprem)
BlendHyderabad, Republic Of India, IN Senior Data Engineers / Data Engineers.Python / Scala, and modern data engineering tools to help design and implement an end-to-end data architecture for a leading enterprise client.This role is part o...Show moreLast updated: 16 days ago
- Promoted
- New!
Firstwave Technologies - Lead Data Engineer - Snowflake
FIRSTWAVE TECHNOLOGIES PRIVATE LIMITEDHyderabad Description : Job Summary : We are seeking an experienced Lead Snowflake Data Engineer to join our Data &...Show moreLast updated: 21 hours ago
- Promoted
Lead Data Engineer
Accordion IndiaHyderabad, Telangana, India Company Overview Accordion is a global private equity-focused financial consulting firm specializing in driving value creation through services rooted in Data & Analytics and powered by technology...Show moreLast updated: 14 days ago
- Promoted
LEAD DATA ENGINEER
Prophecy Technologieshyderabad, telangana, in We’re Hiring : LEAD DATA ENGINEER.Notice Period : Immediate to 30 Days.Design, build, and optimize scalable data pipelines on Azure Data Lake, Azure Databricks, and Azure Synapse.Develop ETL / ELT work...Show moreLast updated: 1 day ago
- Promoted
Tredence Analytics Solutions - Lead Data Engineer - Google Cloud Platform
Tredence Analytics Solutions Private LimitedHyderabad Job Description : Job Title : Lead GCP Data Engineer Skills required : GCP DE Experience, Big query, SQL, Cloud compressor / Python, Cloud functions, Datapro...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Platform Engineer
Black Dog LabsHyderabad, IN Remote (collaboration across time zones), India or LATAM preferred.Proficient English communication.Data Engineering / Backend Engineering / DevOps.
We’re looking for a hands-on Senior Data Platform...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Insight GlobalHyderabad, IN GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts.
Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago