Big Data Engineer (Hadoop)

Key Responsibilities :

• Design, develop, and optimize large-scale data processing workflows using Hadoop components such as HDFS, MapReduce, Hive, Pig, and HBase .
• Build and maintain ETL pipelines to ingest and transform data from various sources into Hadoop clusters.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality data solutions.
• Ensure data quality, integrity, and security within big data environments.
• Monitor and troubleshoot Hadoop cluster performance and resolve issues proactively.
• Implement best practices for data storage, processing, and retrieval in Hadoop ecosystems.
• Develop automation scripts for data pipeline orchestration and workflow management (using tools like Apache Oozie or Airflow).
• Participate in capacity planning, cluster management, and Hadoop upgrades.
• Document data architecture, workflows, and operational procedures.

Qualifications and Requirements :

Skills Required

Java, Scala, Python, Linux, Git