Big Data Engineer

ConfidentialKanpur, India

9 days ago

Job description

Job Summary

We are hiring an experienced Big Data Engineer to design, build, and maintain high-performance data processing systems. The role requires strong expertise in Hadoop ecosystem, Spark, and cloud-based data pipelines with a focus on scalability, reliability, and performance.

Key Responsibilities

Design and develop end-to-end big data pipelines for batch and real-time processing.
Work with large datasets using Spark, Hadoop, Hive, HBase, and Kafka.
Develop data ingestion, transformation, and storage layers.
Optimize data workflows for speed, cost, and efficiency.
Collaborate with analysts, data scientists, and application teams for data delivery.
Ensure data security, lineage, and quality across the platform.
Deploy and manage workloads on AWS, Azure, or GCP.

Required Skills

Proficiency in Spark (PySpark / Scala), Hadoop, Hive, Kafka, HBase, and Sqoop.

Strong programming skills in Python, Scala, or Java.

Experience with ETL / ELT pipelines and data modeling.

Good knowledge of cloud data platforms (AWS EMR, Dataproc, Databricks, Azure HDInsight).

Familiar with data lake and data warehouse integration.

Proficient in SQL and performance tuning.

Knowledge of CI / CD, Docker, and Kubernetes preferred.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, or related field.

Preferred

Cloud certifications (AWS Big Data, GCP Data Engineer, or Azure Data Engineer).

Experience in streaming data pipelines using Kafka / Flink / Spark Streaming.

Exposure to machine learning pipelines and data governance frameworks.

Skills : data,cloud,big data,kafka,spark,hadoop,pipelines,aws,design,azure

Skills Required

Java, Hadoop, Scala, Kafka, Hbase, Sql, ELT, Hive, Gcp, Docker, Sqoop, Spark, Azure, Python, Kubernetes, Aws, Etl

Create a job alert for this search

Big Data Engineer • Kanpur, India