Data Pipeline Engineer

Job Description

We are looking for a Python & PySpark developer and data engineer who can design and build solutions for one of our Fortune 500 Client programs in the realm of Financial Master & Reference Data Management. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.

Key Responsibilities

• Ability to design, build and unit test applications on Spark framework on Python.
• Build Python and PySpark based applications based on data in both Relational (Oracle, SQL Server) and NoSQL database e.G. Redis, Valkey
• Build ingestion pipelines to load data into in-memory caching database.
• Build data pipelines to load and export data to / from the Master Data Management or Reference Data Management Platform using PySpark
• Hands on experience doing at least one MDM implementation using TIBCO, Informatica, GoldenSource or custom-built MDM / RDM platforms
• Experience with designing / building Data APIs and its interaction with data consumers
• Optimize performance of the built Spark applications using configurations around Spark Context, Spark-SQL and Data Frame.
• Build Python programs using module libs e.G. pandas, requests, json, flask, pickle
• Experience in processing large amounts of structured data, including integrating data from multiple sources.
• Ability to design & build real-time applications using REST API, JSON, XML
• Ability to build solutions on AWS services using Glue ETL, Lambda functions on Python
• Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and / or GIT repositories
• Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
• Work collaboratively with onsite and offshore team.
• Develop & review technical documentation for artifacts delivered.
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Participate in code release and production deployment.
• Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

Requirements