DataZymes - Databricks Platform Engineer - ETL

Role Overview :

Seasoned Databricks Senior Platform Engineer who can implement scalable, end-to-end data solutions on Databricks over AWS. This role is pivotal in building the foundational infrastructure for data products, ensuring data quality, and enabling machine learning capabilities

Key Responsibilities :

• Design and implement end-to-end ETL pipelines in Databricks, sourcing data from Amazon S3 and Amazon RDS and other sources.
• Design and deploy Databricks workspaces and clusters optimized for performance, scalability, and cost-efficiency.
• Set up data quality frameworks and anomaly detection mechanisms to ensure reliability and trust in data products.
• Enable machine learning capabilities within Databricks by configuring necessary services, libraries, and integrations.
• Collaborate with data scientists, engineers, and DevOps teams to ensure seamless platform operations.
• Automate deployment and monitoring using CI / CD pipelines, GitHub
• Document architecture, workflows, and operational procedures for platform governance.

Requirements :

Core Platform & Cloud :

ETL & Data Quality :

ML Platform Enablement :

DevOps & Automation :

Preferred Qualifications :

(ref : hirist.tech)