Talent.com
Senior Data Pipeline Engineer

Senior Data Pipeline Engineer

airtelPune, Republic Of India, IN
5 days ago
Job description

Job profiles - Big Data Developer (Senior Manager)

Experience - 7-10 yrs

Location - Pune

Job Description

We are seeking a highly skilled Lead Big Data Developer to architect, drive, and optimize large-scale data engineering solutions in a modern, modular, and highly automated environment. The ideal candidate will have extensive experience in building and leading end-to-end big data pipelines, with deep expertise in distributed systems, data transformation frameworks (such as DBT), and advanced data engineering best practices.

Key Responsibilities

  • Technical Leadership :
  • Lead the design, development, and deployment of big data solutions using Java, Apache Spark, and modern orchestration tools.
  • Architect and optimize data workflows, ensuring modularity, scalability, and maintainability across multiple domains and business units.
  • Guide the adoption and integration of low-code transformation platforms (e.G., DBT) and ensure best practices in SQL-based transformation logic.
  • Should be able to debug and fine tune workload, processing billions of records
  • Solution Delivery & Collaboration :
  • Oversee the development and management of APIs for data ingestion, transformation, and consumption, ensuring seamless integration with upstream and downstream systems.
  • Collaborate with product managers, data scientists, analysts, and platform teams to translate business requirements into technical deliverables.
  • Drive the implementation of metadata-first strategies, ensuring comprehensive cataloging, lineage, and data quality across the platform.
  • Quality, Observability & Governance :
  • Implement and enforce data engineering standards for quality, integrity, and security.
  • Leverage observability suites for active and passive data validations, anomaly detection, and performance monitoring.
  • Ensure robust governance by supporting RBAC / ABAC controls and compliance requirements.
  • Mentorship & Team Development :
  • Mentor and coach a team of big data engineers, fostering a culture of innovation, continuous improvement, and technical excellence.
  • Conduct code reviews, knowledge sharing sessions, and technical upskilling initiatives.
  • Continuous Improvement :
  • Stay abreast of emerging trends in big data, cloud-native technologies, MLOps, and GenAI integration.
  • Drive proof-of-concept (POC) initiatives and evaluate new tools and frameworks for adoption.

Required Qualifications :

  • Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.
  • 8+ years of experience in big data engineering, with at least 2 years in a technical leadership or lead developer role.
  • Advanced proficiency in Java, Apache Spark, and distributed data processing frameworks.
  • Hands-on experience with data transformation tools such as DBT, and orchestration platforms (Airflow, Nifi, etc.).
  • Strong expertise in API development, integration, and performance tuning.
  • Deep understanding of data engineering principles, data mesh concepts, and multi-domain data architectures.
  • Proficiency with Hadoop ecosystem tools (HDFS, Hive, HBase, Kafka, etc.).
  • Experience with SQL and NoSQL databases.
  • Strong problem-solving, analytical, and communication skills.

    • Preferred Qualifications

  • Experience with cloud platforms (AWS, Azure, GCP), Spark / Delta Lake, and modern data lake architectures.
  • Knowledge of containerization (Docker, Kubernetes) and CI / CD pipelines.
  • Familiarity with MLOps, GenAI integration, and machine learning frameworks.
  • Experience in implementing metadata management, data lineage, and observability solutions.
  • Exposure to federated governance, RBAC / ABAC, and security best practices.

    • Benefits

  • Competitive salary and comprehensive benefits package.
  • Opportunity to lead and shape the next-generation data engineering platform in a highly collaborative environment.
  • Access to cutting-edge technologies and continuous professional development.
  • Work with a diverse, talented team on impactful, large-scale data initiatives.

    • If you are passionate about leading technical teams, architecting scalable data solutions, and driving innovation in a fast-paced environment, we invite you to join us as our Lead Big Data Developer.

    Create a job alert for this search

    Senior Data Engineer • Pune, Republic Of India, IN