Technical Skills
- Databricks Lakehouse Platform :
o Medallion Architecture o Unity Catalog (data governance, lineage)
o Delta Lake & DLT Pipelines
o PySpark Workbooks o Spark SQL & SQL Warehouse
Programming : Python, SQL, PySparkAWS Cloud Services : IAM, S3, Lambda, EMR, Redshift, BedrockOther :o Familiarity with DevOps and CI / CD processes
o Experience with any Data Security tool
Key Responsibilities
Establish, enforce, and monitor Data Engineering governance standards, best practices, and guidelines across enterprise and self-service environments.Develop and maintain documentation for data engineering processesDefine and implement data security policies and role-based access controls (RBAC / ABAC) across all data engineering processes.Oversee data classification, lifecycle management, and comprehensive metadata management to ensure transparency, traceability, and compliance.Implement and manage robust change control processes for all data engineering activitiesMonitor, maintain, and optimize data pipelines and workflows, ensuring reliability, scalability, and efficiency.Continuously monitor and optimize the performance of data engineering processes and resource utilization.Promote a culture of data engineering governance, data security awareness, and operational excellence within the team and across the organization.