Lead Data Engineer Databricks, AWS

Description

We have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.

As a Lead Software Engineer at JPMorgan Chase within the Commercial & Investment Banking Data Analytics Payments Technology team youare an integral part of an agile team that works to enhance build and deliver trusted market-leading technology products in a secure stable and scalable way. As a core technical contributor you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

Job responsibilities

• Executes creative software solutions design development and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
• Designs builds and maintains scalable data pipelines and ETL/ELT workflows for batch and real-time processing using Spark Airflow Kafka and Flink
• Develops data platform components including data cataloging data quality frameworks and semantic/metrics layers with embedded governance lineage and compliance standards
• Implements data modeling strategies (fact and dimensional wide tables) to support analytics reporting and downstream consumption
• Partners with analytics teams product managers and business stakeholders to translate data requirements into production-grade solutions
• Develops secure high-quality production code and reviews and debugs code written by others
• Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems
• Leads evaluation sessions with external vendors startups and internal teams to drive outcomes-oriented probing of architectural designs technical credentials and applicability for use within existing systems and information architecture

• Leads communities of practice across Software Engineering to drive awareness and use of new and leading-edge technologies

Required qualifications capabilities and skills

• Formal training or certification on software engineering concepts and 5 years of applied experience
• Hands-on practical experience delivering system design application development testing and operational stability
• Demonstrated professional experience focused on software engineering or data platform development
• Experience designing and operating data pipelines on Databricks using orchestration tools such as Apache Airflow
• Proficiency with cloud data services (AWS S3 Glue Redshift Athena EMR Lake Formation or equivalent)
• Advanced in Python and SQL programming languages
• Hands-on experience with distributed data processing frameworks such as Apache Spark and Flink

• Solid understanding of data modeling techniques (star schema snowflake) and query optimization

  )

• Experience engineering production-grade data platforms on Kubernetes with open catalog integration (e.g. Apache Iceberg Unity Catalog OpenMetadata) for scalable data discovery lineage and governance.
• Advanced understanding of agile methodologies such as CI/CD Application Resiliency and Security
• Demonstrated proficiency in software applications and technical processes within a technical discipline (e.g. cloud artificial intelligence machine learning mobile etc.)
  

Preferred qualifications capabilities and skills

• Experience with Agentic AI LLMs RAG architectures vector databases and embedding-based retrieval systems
• Hands-on familiarity with Internal Developer Portals such as Backstage including service catalog management software templating and plugin development
• Experience with data mesh or data product architectures
• Proficiency with Infrastructure as Code (Terraform) and containerized deployments (Docker Kubernetes)
• Experience with data observability quality and metadata management tools
• Experience with semantic layers metrics stores or BI platforms (Tableau dbt Metrics)

Required Experience:

IC

Employment Type : Full-Time
Experience: years
Vacancy: 1