Data Engineering Expert

Job Title : Data Engineering Lead

Data Engineering Lead job summary :

Lead a team of big data engineers to design and implement scalable architectures that meet client needs.

Key Responsibilities :

The Data Engineering Lead will lead the development of new data pipelines, data transformations within the Data Lake, implementing GraphQL, working on NoSQL databases, CI / CD, and data delivery as per business requirements. They will also collaborate with cross-functional teams to source data and make it available for downstream consumption.

Responsibilities include managing dependencies and challenges, escalating critical issues to the Sponsor and / or Head of Data Engineering. The Data Engineering Lead must ensure timelines are managed and value of initiative is achieved without compromising quality and within budget.

The Data Engineering Lead will be responsible for implementing coordinated communications plans for initiative execution and delivery, both internal and external. This includes final handover of initiatives to business-as-usual processes and conducting post-implementation reviews to ensure initiative objectives have been delivered and lessons learned are fed into future initiative management processes.

Required Skills and Qualifications :

The ideal candidate will have experience in building data pipelines using batch processing with Apache Spark or Hive query language. Experience in building streaming data pipelines using Apache Spark Structured Streaming or Apache Flink on Kafka & Delta Lake is also required. Knowledge of NoSQL databases, big data ETL processing tools, data modeling, and data mapping is essential.

The successful candidate will have a basic understanding of CI / CD tools, DevOps practices using Git version control, and the ability to debug, fine-tune, and optimize large-scale data processing jobs. Additionally, they will have experience with Hive and Hadoop file formats, basic knowledge of scripting (shell / bash), and experience working with multiple data sources including relational databases, NoSQL / document databases, and flat files.