Lead AI Platform Engineer

JOB DESCRIPTION

Lead Databricks Data Engineer

Ecolab is looking for a Data Engineer to be part of a dynamic team that's at the forefront of technological innovation. We're leveraging cutting-edge AI to create novel solutions that optimize operations for our clients, particularly within the restaurant industry. Our work is transforming how restaurants operate, making them more efficient and sustainable.

As a key player in our new division, you'll have the unique opportunity to help shape its culture and direction. Your contributions will directly impact the success of our innovative projects and help define the future of our product offerings. Additionally, you will experience the best of both worlds with this team at Ecolab : the agility and creativity of a startup paired with the stability and resources of a global leader. Our collaborative environment fosters innovation while providing the support and security you need to thrive.

Responsibilities

• Design, develop, and maintain scalable and robust data pipelines on Databricks (Spark SQL, PySpark, Delta Lake).
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
• Optimize and troubleshoot existing data pipelines for performance and reliability.
• Ensure data quality and integrity across various data sources.
• Implement data security and compliance best practices.
• Monitor data pipeline performance, implement data quality checks, and conduct necessary maintenance and updates.
• Document data pipeline processes and technical specifications.
• Implement robust pipeline orchestration using tools like Databricks Workflows, dbt, or similar.
• Generate and maintain data quality reports and dashboards.
• Implement Infrastructure as Code (IaC) principles for managing Databricks infrastructure.

Minimum Qualifications

Preferred Qualifications

Our Commitment to a Culture of Inclusion & Belonging

Ecolab is committed to fair and equal treatment of associates and applicants and furthering the principles of Equal Opportunity to Employment. We will recruit, hire, promote, transfer and provide opportunities for advancement based on individual qualifications and job performance in all matters affecting employment, compensation, benefits, working conditions, and opportunities for advancement. Ecolab will not discriminate against any associate or applicant for employment because of race, religion, color, creed, national origin,citizenship status, sex, sexual orientation, gender identity and expressions, genetic information, marital status, age, or disability.