Senior ETL & Data Streaming Engineer

About the Organisation :

DataFlow Group is a pioneering global provider of specialized Primary Source Verification (PSV) solutions, and background screening and immigration compliance services that assist public and private organizations in mitigating risks to make informed, cost-effective decisions regarding their Applicants and Registrants.

About the Role :

We are looking for a highly skilled and experienced Senior ETL & Data Streaming Engineer with over 10 years of experience to play a pivotal role in designing, developing, and maintaining our robust data pipelines.

The ideal candidate will have deep expertise in both batch ETL processes and real-time data streaming technologies, coupled with extensive hands-on experience with AWS data services. A proven track record of working with Data Lake architectures and traditional Data Warehousing environments is essential.

Duties and Responsibilities :

• Design, develop, and implement highly scalable, fault-tolerant, and performant ETL processes using industry-leading ETL tools to extract, transform, and load data from various source systems into our Data Lake and Data Warehouse.
• Architect and build batch and real-time data streaming solutions using technologies like Talend, Informatica, Apache Kafka or AWS Kinesis to support immediate data ingestion and processing requirements.
• Utilize and optimize a wide array of AWS data services
• Collaborate with data architects, data scientists, and business stakeholders to understand data requirements and translate them into efficient data pipeline solutions.
• Ensure data quality, integrity, and security across all data pipelines and storage solutions.
• Monitor, troubleshoot, and optimize existing data pipelines for performance, cost-efficiency, and reliability.
• Develop and maintain comprehensive documentation for all ETL and streaming processes, data flows, and architectural designs.
• Implement data governance policies and best practices within the Data Lake and Data Warehouse environments.
• Mentor junior engineers and contribute to fostering a culture of technical excellence and continuous improvement.
• Stay abreast of emerging technologies and industry best practices in data engineering, ETL, and streaming.

Qualifications :

1. Proficiency with AWS S3 for data storage and management.

2. Hands-on experience with AWS Glue for ETL orchestration and data cataloging.

3. Familiarity with AWS Lake Formation for building secure data lakes.

4. Good to have experience with AWS EMR for big data processing

(ref : hirist.tech)