Position : Big Data Engineer
Experience : 59 Years
Job Type : Full-time
Job Summary :
We are seeking a highly skilled Big Data Engineer with 59 years of experience to develop, maintain, and optimize robust, scalable big data solutions. This role requires expertise in designing and implementing solutions using Hadoop-based technologies and AWS. The engineer will be crucial in building various complex Big Data platform components, focusing on optimizing data access, handling high data volumes, and ensuring real-time performance using frameworks like Spark, Kafka, and other cloud technologies.
Key Responsibilities :
Big Data Platform Architecture and Design :
- Lead the design of comprehensive data solutions leveraging Hadoop-based technologies in conjunction with AWS cloud infrastructure.
- Implement and design various critical Big Data platform components, including Batch Processing, Live Stream Processing, In-Memory Cache, Query Layer (SQL), Rule Engine, and Action Framework.
- Design and implement a robust Data Access Layer capable of connecting to diverse data sources.
Performance and Scalability :
Implement scalable technical solutions to reliably handle ever-increasing data volumes using modern big data and cloud technologies like Spark, Kafka, and Cloud computing services.Utilize advanced caching techniques within the Data Access Layer to ensure fast responses to real-time SQL queries against Big Data Technologies.Data Pipeline and Infrastructure :
Set up and operate end-to-end data pipelines and the overall infrastructure required for executing ETL jobs.Apply proficiency with distributed file systems and computing concepts to optimize data :Experience : 59 years of hands-on Data engineering experience.Programming Languages : Good hands-on knowledge and proficiency in one of the primary languages : Java, Scala, or Python.Data Handling Frameworks : Strong hands-on experience with at least one of the major data handling frameworks : Apache Spark, Apache Beam, or Apache Flink.Data Store : Good technical understanding of both SQL and NoSQL databases (e.g., MySQL, Mongo).Data Infrastructure : Familiarity with cloud solutions for data infrastructure, with AWS preferably.Distributed Concepts : Proficient with distributed file systems and computing concepts (e.g., HDFS, YARN).Preferred Skills :
Familiarity with different types of efficient data formats, including Apache Parquet, Avro, and ORC.Direct experience integrating streaming technologies like Kafka with processing frameworks like Spark or Flink.Experience in designing a Rule Engine or Action Framework for real-time data processing.Certifications in AWS Data Analytics or related Big Data technologies.(ref : hirist.tech)