Lead Data Pipeline Engineer

Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services Lead the E-Comms data pipeline within Compass’s Application Simplification workstream : design, build, and optimize AWS-native ingestion and enrichment of high-volume communications (Bloomberg, ICE, LSEG / Refinitiv, Symphony, etc.). Ensure robust retention, security, and data integrity, and deliver reliable, analytics- and AI-ready datasets for downstream platforms.

Role Responsibilities :

• Build AWS infrastructure using Terraform or CDK
• Develop, maintain, test and optimize data pipelines and workflows between the source data points and target destinations.
• Implement data management and governance frameworks to ensure compliance with regulatory and legal requirements, and security protocols.
• Collaborate with other team members such as the Project Manager and Business Analysts to identify and address risks and challenges in real time.
• Write unit, integration, and infrastructure tests
• Monitor, troubleshoot and resolve issues.
• Automate the data pipeline where possible to reduce manual risk and improve efficiency.
• Work closely with stakeholders and subject matter experts to ensure success criterion and definition of done is met.
• Stay updated on industry trends and emerging tech to ensure continuous improvement.

Experience / Competences

Essential :

Desired :

MSBC Group has been a trusted technology partner for over 20 years, delivering the latest systems and software solutions for financial services, manufacturing, logistics, construction, and startup ecosystems. Our expertise includes Accessible AI, Custom Software Solutions, Staff Augmentation, Managed Services, and Business Process Outsourcing.

We are at the forefront of developing advanced AI-enabled services and supporting transformative projects, such as state-of-the-art trading platforms, seamless application migrations, and integrating real-time data analytics. With offices in London, California, and Ahmedabad, and operating in every time-zone, MSBC Group is your AI and automation partner.