Senior Pipeline Engineer

This role is for one of Weekday’s clients

Min Experience : 6 years

Location : India

JobType : full-time

Requirements

We are looking for an experienced and motivated Senior Pipeline Engineer to join our engineering team and take ownership of designing, building, and managing high-performance data pipelines. This role demands strong expertise in orchestration frameworks such as Celery and Temporal , combined with advanced skills in database design and query optimization . The ideal candidate will have a passion for solving complex technical challenges and delivering reliable, scalable solutions that power data-driven decision-making across the organization.

As a senior team member, you will play a critical role in shaping the future of our data infrastructure, working closely with cross-functional teams including data scientists, backend engineers, and product managers. You will also mentor junior engineers and drive best practices in orchestration, database efficiency, and scalable pipeline design.

Key Responsibilities

• Pipeline Development & Management
• Design, develop, and maintain robust and scalable data pipelines for batch and real-time processing.
• Automate workflows for data ingestion, transformation, validation, and delivery.
• Ensure pipelines are fault-tolerant, performant, and cost-efficient .
• Workflow Orchestration
• Build and manage orchestration workflows using Celery, Temporal, or similar tools .
• Monitor, schedule, and optimize pipeline execution to reduce latency and improve reliability.
• Implement scalable solutions for dependency management, retries, and error handling.
• Database Design & Query Optimization
• Create efficient database schemas and models to support large-scale data operations.
• Optimize complex queries, indexing, and partitioning strategies to ensure high performance .
• Collaborate with engineering teams to enforce database best practices and scalability standards.
• Collaboration & Leadership
• Partner with product managers, analysts, and business stakeholders to understand requirements and translate them into technical solutions.
• Mentor and guide junior engineers in pipeline development and orchestration practices.
• Contribute to architecture reviews, design discussions, and technical documentation.
• Continuous Improvement & Compliance
• Proactively identify opportunities to improve pipeline performance and reliability .
• Stay updated on industry trends, orchestration frameworks, and database technologies.
• Ensure adherence to data security, governance, and compliance policies.

Required Skills & Qualifications

Preferred Skills