Talent.com
This job offer is not available in your country.
Senior Big Data Engineer - Hadoop / Kafka / Spark

Senior Big Data Engineer - Hadoop / Kafka / Spark

FxConsultingGurgaon
11 hours ago
Job description

Position : Big Data Engineer

Experience : 59 Years

Job Type : Full-time

Job Summary :

We are seeking a highly skilled Big Data Engineer with 59 years of experience to develop, maintain, and optimize robust, scalable big data solutions. This role requires expertise in designing and implementing solutions using Hadoop-based technologies and AWS. The engineer will be crucial in building various complex Big Data platform components, focusing on optimizing data access, handling high data volumes, and ensuring real-time performance using frameworks like Spark, Kafka, and other cloud technologies.

Key Responsibilities :

Big Data Platform Architecture and Design :

  • Lead the design of comprehensive data solutions leveraging Hadoop-based technologies in conjunction with AWS cloud infrastructure.
  • Implement and design various critical Big Data platform components, including Batch Processing, Live Stream Processing, In-Memory Cache, Query Layer (SQL), Rule Engine, and Action Framework.
  • Design and implement a robust Data Access Layer capable of connecting to diverse data sources.

Performance and Scalability :

  • Implement scalable technical solutions to reliably handle ever-increasing data volumes using modern big data and cloud technologies like Spark, Kafka, and Cloud computing services.
  • Utilize advanced caching techniques within the Data Access Layer to ensure fast responses to real-time SQL queries against Big Data Technologies.
  • Data Pipeline and Infrastructure :

  • Set up and operate end-to-end data pipelines and the overall infrastructure required for executing ETL jobs.
  • Apply proficiency with distributed file systems and computing concepts to optimize data :
  • Experience : 59 years of hands-on Data engineering experience.
  • Programming Languages : Good hands-on knowledge and proficiency in one of the primary languages : Java, Scala, or Python.
  • Data Handling Frameworks : Strong hands-on experience with at least one of the major data handling frameworks : Apache Spark, Apache Beam, or Apache Flink.
  • Data Store : Good technical understanding of both SQL and NoSQL databases (e.g., MySQL, Mongo).
  • Data Infrastructure : Familiarity with cloud solutions for data infrastructure, with AWS preferably.
  • Distributed Concepts : Proficient with distributed file systems and computing concepts (e.g., HDFS, YARN).
  • Preferred Skills :

  • Familiarity with different types of efficient data formats, including Apache Parquet, Avro, and ORC.
  • Direct experience integrating streaming technologies like Kafka with processing frameworks like Spark or Flink.
  • Experience in designing a Rule Engine or Action Framework for real-time data processing.
  • Certifications in AWS Data Analytics or related Big Data technologies.
  • (ref : hirist.tech)

    Create a job alert for this search

    Senior Data Engineer • Gurgaon

    Related jobs
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    EXLDelhi, IN
    We are seeking a highly skilled.You will work closely with cross-functional teams to provide thought leadership deliver robust, secure, and high-performance data solutions that empower analytics an...Show moreLast updated: 26 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    CESDelhi, IN
    As part of our data engineering team, you’ll build and maintain.This is a hands-on role where you’ll design and optimize modern data infrastructure, ensuring reliability, scalability, and performan...Show moreLast updated: 6 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    EvolentDelhi, IN
    Lead the design, development, and deployment of complex data solutions, ensuring scalability and maintainability.Architect and implement high-performance data pipelines and data warehousing solutio...Show moreLast updated: 24 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Eucloid Data SolutionsDelhi, IN
    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 8 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    The Mastermind GroupDelhi, IN
    Remote (India) – Occasional travel to Riyadh, KSA.Sunday–Thursday preferred (9 AM – 5 PM IST).About The Mastermind Group (TMG). Software Asset Management (SAM).Hardware Asset Management (HAM).We're ...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Otomeyt AIGhaziabad, IN
    We are seeking a highly skilled 7+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 18 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    SAIVA AIDelhi, IN
    We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 8 days ago
    • Promoted
    • New!
    Senior Generative AI Engineer (Databricks,Data Lake)

    Senior Generative AI Engineer (Databricks,Data Lake)

    AmpstekDelhi, IN
    Title : Senior Generative AI Engineer (Databricks,Data Lake).Design and implement GenAI models (LLMs, multimodal, embeddings, and fine-tuning) for enterprise use cases. Architect and optimize data pi...Show moreLast updated: 20 hours ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    DhurinGreater Delhi Area, India
    Dhurin is a fastgrowing data science firm serving financial services industry.We work with banks, NBFCs, fintech and other financial institutions. Our work is in the space of risk, fraud, collection...Show moreLast updated: 27 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    HealthyMe AiDelhi, IN
    We are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data.You will report to the Head of AI and Engineering and be responsible for designi...Show moreLast updated: 26 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    DeltacubesDelhi, IN
    Build and maintain scalable ETL / ELT pipelines.Work with Snowflake and BigQuery for data storage.Implement orchestration with Airflow or Prefect. Integrate data workflows with Python.Optimize data pi...Show moreLast updated: 15 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    DexianDelhi, IN
    We seek a Senior Data Engineer colleague to join our world-class client, one of the.Employment / Contract Type : Contract. Duration : 1 year (with potential to be extended).Hours per Week : 40 (Full-time...Show moreLast updated: 26 days ago
    • Promoted
    • New!
    Senior Generative AI Engineer (Databricks Data Lake)

    Senior Generative AI Engineer (Databricks Data Lake)

    AmpstekDelhi, IN
    Title : Senior Generative AI Engineer (Databricks Data Lake).We are seeking an experienced Senior Generative AI Engineer with a strong background in Databricks and data lake architectures.This indiv...Show moreLast updated: 20 hours ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    InfogainDelhi, IN
    Big Data Engineer (Lead) : As a Big Data Engineer (Lead), you will be responsible for leading a team of big data engineers. You will work closely with clients and team members to understand their req...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Otomeyt AIDelhi, IN
    We are seeking a highly skilled 5+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    HISH IT SERVICESDelhi, IN
    Location : Remote(Banglore,Chennai,Pune).Pay : 14LPA - 18 LPA(Based on Experience).Timings : A couple of hours overlap with EST, as the client is Canada-based (till 12AM IST).Start Date : 20th Octob...Show moreLast updated: 7 days ago
    • Promoted
    Senior Big Data Engineer

    Senior Big Data Engineer

    VeltrisDelhi, IN
    Veltris is a Digital Product Engineering Services partner committed to driving technology-enabled transformation across enterprises, businesses, and industries. We specialize in delivering next-gene...Show moreLast updated: 15 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    BrightEdgeDelhi, IN
    BrightEdge About Us BrightEdge is a leading SEO and content performance marketing platform that transforms online content into tangible business results. Our platform processes massive amounts of da...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    Senior Data Engineer

    Senior Data Engineer

    Net Zero InsightsDelhi, IN
    Net Zero Insights is a leading provider of market intelligence for the Net-Zero Economy, offering comprehensive insights, data, and research to climate innovation professionals worldwide.Our cuttin...Show moreLast updated: 20 hours ago
    • Promoted
    Data Engineer

    Data Engineer

    Innodata Inc.Delhi, IN
    CI / CD practices, Databricks (Spark), Python, Github and SQL.The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and...Show moreLast updated: 26 days ago