Data Engineer – Big Data Platforms
Job Summary :
We are seeking a skilled and experienced Data Engineer to join our team. The ideal candidate will have expertise in Big Data platforms and a strong background in designing, developing, and maintaining data pipelines. You will be responsible for ensuring data quality, optimizing data workflows, and working with distributed data systems. This role requires a deep understanding of modern data engineering workflows, ETL processes, and large-scale data processing.
Key Responsibilities :
Data Engineering :
- Design, develop, and maintain scalable and efficient data pipelines to support business needs.
- Implement ETL / ELT processes to extract, transform, and load data from various sources into Big Data platforms.
- Optimize data workflows for performance, scalability, and reliability.
Big Data Platforms :
Work with Big Data technologies such as Hadoop, Spark, Hive, or HDFS to process and analyze large datasets.Develop distributed data processing pipelines using Spark (PySpark / Scala) and MapReduce.Optimize Big Data workflows for performance and scalability.Data Integration :
Integrate data from multiple sources, including APIs, flat files, and third-party systems.Collaborate with data analysts, data scientists, and business teams to understand data requirements.Data Quality and Governance :
Ensure data accuracy, consistency, and integrity across systems.Implement data validation and error-handling mechanisms.Collaboration and Documentation :
Work closely with cross-functional teams to understand business requirements and translate them into technical solutions.Document data pipelines, data designs, and processes for future reference.Required Skills and Qualifications :
Technical Expertise : Expertise in Big Data platforms : Experience with Hadoop, Spark (PySpark / Scala), Hive, or HDFS for distributed data processing.Proficiency in SQL and database design principles.Experience with ETL / ELT tools (, Informatica, Talend, or Python-based ETL frameworks).Programming Skills : Proficiency in scripting languages such as Python , Scala , or Shell scripting for data processing and automation.Data Governance : Knowledge of data quality, data governance, and data security best practices.Soft Skills : Strong problem-solving and analytical skills.Excellent communication and collaboration skills.Ability to work in a fast-paced, dynamic environment.
Qualifications :
7-9 years of relevant experienceExperience in systems analysis and programming of software applicationsExperience in managing and implementing successful projectsWorking knowledge of consulting / project management techniques / methodsAbility to work under pressure and manage deadlines or unexpected changes in expectations or requirementsEducation :
Bachelor’s degree / University degree or equivalent experienceThis job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required.
Job Family Group : Technology
Job Family :
Applications Development
Time Type : Full time
Most Relevant Skills
Please see the requirements listed above.
Other Relevant Skills
For complementary skills, please see above and / or contact the recruiter.