Senior Data Engineer

We are looking for Senior Data Engineers who have the right attitude, aptitude, skills, empathy, compassion, and hunger for learning. Build products in the data analytics space. A passion for shipping high-quality data products, interest in the data products space; curiosity about the bigger picture of building a company, product development and its people.

Roles and Responsibilities

• Develop and manage robust ETL pipelines using Apache Spark (Scala)
• Understand Spark concepts, performance optimization techniques, and governance tools
• Develop a highly scalable, reliable, and high-performance data processing pipeline to extract, transform, and load data from various systems to the Enterprise Data Warehouse / Data Lake / Data Mesh hosted on AWS or Azure
• Collaborate cross-functionally to design effective data solutions
• Implement data workflows utilizing AWS Step Functions or Azure Logic Apps for efficient orchestration. Leverage AWS Glue and Crawler or Azure Data Factory and Data Catalog for seamless data cataloging and automation
• Monitor, troubleshoot, and optimize pipeline performance and data quality
• Maintain high coding standards and produce thorough documentation. Contribute to high-level (HLD) and low-level (LLD) design discussions.

Technical Skills