Kaleidofin - Lead Data Engineer - ETL / PySpark

What youll do ?

• You will take on the role of technical expert and a lead member of our Engineering team.
• Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc.
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
• Designing, building, and operationalizing large-scale enterprise data solutions using Hadoop based technologies along with AWS, Spark, Hive, & Data Lake, using Hive, PySpark and Python Programming.
• Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various use cases built on the platform.
• Responsible to Ingest data from files, streams and databases and process the data with Hadoop,Scala, SQL / NOSQL Database, Spark, ML, IoT.
• Develop programs in Scala and Python as part of data cleaning and processing.
• It includes data modelling, data ingestion, transformation, data consumption patterns and optimizing complex queries, creating efficient UDFs to extend the functionalities.
• You will be involved in product feature development and will be working in close partnership with other engineering teams.
• You will be responsible for mentoring other team members and ensuring high availability platform and stability for batch and stream processing systems.

Who you need to be ?

Spark, Hive, Kafka etc.

(ref : hirist.tech)