Lead Data Engineer - Python / SQL / ETL

Job Description :

We are seeking a highly skilled Lead Data Engineer to join our data engineering team for an on-premise environment. A large portion of your time will be in the weeds working alongside your team on architecture, designing, implementing, and optimizing data solutions. The ideal candidate will have extensive experience in building and optimizing data pipelines, architectures, and data sets, with a strong focus on Python, SQL, Hadoop, HDFS, and Apache NiFi.

Responsibilities :

• Design, develop, and maintain robust, scalable, and high-performance data pipelines and data integration solutions.
• Manage and optimize data storage in Hadoop Distributed File System (HDFS).
• Design and implement data workflows using Apache NiFi for data ingestion, transformation, and distribution.
• Collaborate with cross-functional teams to understand data requirements and deliver efficient solutions.
• Ensure data quality, governance, and security standards are met within the on-premise infrastructure.
• Monitor and troubleshoot data pipelines to ensure optimal performance and reliability.
• Automate data workflows and processes to enhance system efficiency.

Requirements :

Good to Have :

ref : hirist.tech)