Big Data Engineer - Spark / HDFS

Description :

Experience : Level 3 : 6-8 Years of exp

Location : Hyderabad

Skill : Python, Spark, HFDS, MongoDB

About the Role :

We are seeking a highly skilled Data Engineer to join our team to design, build, and optimize scalable data pipelines and platforms.

The ideal candidate will have hands-on experience with Python, Spark, HDFS, and MongoDB, and a proven ability to work with large-scale datasets in a distributed Responsibilities :

• Design, develop, and maintain end-to-end data pipelines for batch and real-time processing.
• Work with Apache Spark to process and transform large datasets efficiently.
• Manage and optimize HDFS storage, ensuring data availability, reliability, and performance.
• Develop scripts and data orchestration workflows using Python.
• Build and maintain NoSQL data solutions using MongoDB, including data modeling and performance tuning.
• Collaborate with Data Scientists, Analysts, and Platform Engineering teams to deliver high-quality data solutions.
• Implement data quality, validation, and monitoring frameworks to ensure accuracy and consistency.
• Participate in design reviews, code reviews, and performance optimization initiatives.
• Contribute to the continuous improvement of data engineering standards and best practices.

Required Skills & Qualifications :

Good to Have (Optional) :

(ref : hirist.tech)