Lead Data Engineer - PySpark/Python

Lead Data Engineer role based in Mumbai (Lower Parel). This is a Work From Office position and requires strong expertise in AWS Data Engineering, team leadership, and client-facing responsibilities.

Location : Lower Parel, Mumbai (Work From Office)

Employment Type : Full-Time

Experience Required : 7+ Years

Job Summary :

We are seeking a technically strong and delivery-focused Lead Engineer to support and enhance enterprise-grade data and application products under the Durables model. The ideal candidate will act as the primary technical interface for the client, ensuring high system availability, performance, and continuous improvement. This role requires a hands-on technologist with strong team management experience, cloud (AWS) expertise, and excellent communication skills to handle client interactions and drive technical decisions.

Key Responsibilities :

Support & Enhancement Leadership :

- Act as the primary technical lead for support and enhancement of assigned products in the Durable portfolio.

- Ensure incident resolution, problem management, and enhancement delivery within agreed SLAs.

- Perform root cause analysis (RCA) and provide technical solutions to recurring issues.

- Should design data engineering solutions end to end with the ability to come up with scalable and modular solutions.

- Experience working in Agile implementations.

Technical Ownership :

- Provide technical direction and architectural guidance for improvements, optimizations, and issue resolutions.

- Drive best practices in code performance tuning, ETL processing, and cloud-native data management.

- Lead the modernization of legacy data pipelines and applications by leveraging AWS services (Glue, Lambda, Redshift, S3, EMR, Athena, etc.).

- Leverage PySpark, SQL, Python, and other tools to manage big data processing pipelines efficiently.

Client Engagement :

- Maintain high visibility with client stakeholders, act as a trusted technical advisor.

- Proactively identify and suggest improvements or innovation areas to improve business outcomes.

- Participate in daily stand-ups, retrospectives, and client presentations; communicate technical concepts clearly.

Team & Delivery Management :

- Lead a cross-functional team of engineers, ensuring effective task allocation, mentorship, and upskilling.

- Monitor team performance, support capacity planning, and ensure timely and high-quality deliveries.

- Ensure adherence to governance, documentation, change management practices and High Availability.

Process & Quality Assurance :

- Implement and ensure compliance with engineering best practices including CI/CD, version control, and automated testing.

- Define support procedures, documentation standards, and ensure knowledge transition and retention within the team.

- Identify risk areas, dependencies, and propose mitigation strategies.

Required Skills & Qualifications :

- 8+ years of experience in Data Engineering / Application Support and Development.

- 4+ years of strong hands-on expertise in AWS ecosystem (Glue, Lambda, Redshift, S3, Athena, EMR, CloudWatch).

- Proficiency in PySpark, SQL, Python, and handling big data pipelines.

- Strong application debugging skills across batch and near real-time systems.

- Good knowledge of incident lifecycle management, RCA, and performance optimization.

- Proven experience leading engineering teams (3+ years), preferably in support/enhancement environments.

- Excellent communication skills with proven client-facing capabilities.

- Strong documentation and process adherence mindset.

- Experience with tools like JIRA, Confluence, Git, Jenkins, or any CI/CD pipeline.

Good to Have :

- Experience in on-prem to AWS migration projects.

- Familiarity with legacy tech and their interaction with modern cloud stacks.

- Good knowledge of designing Hive tables with partitioning for performance.

(ref:hirist.tech)