Data Engineer (Lead) ID42105

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI / ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US

If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

As a Lead Data Engineer (Python) , you'll drive the development and optimization of data workflows, ensuring scalable, efficient, and high-quality solutions across Python and Spark environments. You'll take end-to-end ownership of features, collaborate with global stakeholders, and proactively improve processes, gaining exposure to advanced data engineering, workflow automation, and innovative financial systems within an agile, dynamic team.

WHAT YOU WILL DO

• Provide BAU (Business-As-Usual) support for existing data products;- Redesign and optimize existing workflows;
• Implement enhancements and new features using Python / Spark and React;
• Maintain clean, scalable, and well-documented code;
• Automate repetitive production scripts for BAU and remediation;
• Build dashboards tracing data flow from ingestion to distribution;
• Support release readiness activities (dry runs, pre-release validation, regression runs);
• Conduct post-release verification with business stakeholders;
• Suggest and implement platform and process improvements proactively;
• Document all work in Jira and Confluence;
• Participate in Agile ceremonies and scrums;
• Support knowledge transfer within the team.

MUST HAVES

NICE TO HAVES

PERKS AND BENEFITS

Skills Required

Airflow, React, Pandas, dbt, Pyspark, Pytest, Apache Spark, Databricks, Python