Lead Data Engineer (AWS) | Lead Data Engineer | BFSI / Data & Analytics | 6–12 years | Noida | Max

About the Role

We are looking for an experienced Lead Data Engineer to design, build, and optimize cloud-native data platforms on AWS. You will lead a team of Data Engineers and drive end-to-end development of scalable data pipelines, modern data architectures, and enterprise-grade data solutions for BFSI customers.

This is a hands-on role with strong ownership across architecture, development, optimisation, and team leadership.

Key Responsibilities

Leadership & Delivery

Lead, mentor, and manage a team of Data Engineers to deliver high-quality outcomes.

Own project planning, task delegation, sprint delivery, and quality assurance.

Collaborate with cross-functional teams including data scientists, analysts, architects, and product teams.

Data Engineering & Architecture

Architect, design, and implement scalable data pipelines using AWS, PySpark, and Airflow .

Build and maintain ETL / ELT processes for ingestion, transformation, and publishing of structured / unstructured data.

Develop scalable data models and cloud-native architectures aligned with BFSI requirements.

Optimize data processing (Spark), data storage (S3, Iceberg / Hudi / Delta), and query performance.

Cloud & Platform Engineering

Work across AWS services such as S3, Glue, EMR, Lambda, Redshift, Kinesis, RDS, CloudWatch, IAM.

Ensure cost-efficiency, reliability, and scalability of the cloud data environment.

Implement automation for CI / CD, data deployments, and pipeline orchestration.

Governance, Security & Compliance

Enforce data governance standards, lineage tracking, and data quality frameworks.

Implement data security, encryption, RBAC, and compliance aligned with BFSI / regulatory norms.

Ensure high availability, monitoring, and proactive issue resolution.

Required Skills & Experience

Core Technical Skills

6+ years of hands-on experience in data engineering.

Strong expertise in PySpark for distributed data processing.

Experience building and orchestrating pipelines using Airflow (DAGs, scheduling, monitoring).

Strong AWS experience with focus on data services.

Advanced SQL skills and experience in OLTP, OLAP, Data Lakes, and Data Warehouses.

Experience with Hudi / Iceberg / Delta Lake .

Proficiency in Python for ETL, utilities, and API integrations.

Understanding of big data technologies (Spark, Hadoop ecosystem).

Preferred Skills

Experience in BFSI (Banking, Insurance, NBFCs, AMCs) data ecosystems.

Exposure to Docker, Kubernetes, and containerized data applications.

API development experience (preferably Python / FastAPI / Flask).

Experience working with data scientists and BI teams.

Behavioral & Leadership Skills

Strong ownership mindset and accountability.

Excellent problem solving and debugging skills.

Ability to mentor junior team members and lead by example.

Strong communication and stakeholder management.

Ability to work in a fast-paced, entrepreneurial environment.

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.