Data Pipeline Specialist

Atomic NorthPune, Republic Of India, IN

7 days ago

Job description

Big Data Engineer

Location : Pune / Nagpur (WFO)

Experience Required : 8+ Years

Long term onsite to Abu Dhabi

Job Summary :

We are looking for an experienced Big Data Engineer with strong expertise in PySpark and Big Data ecosystems.

The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high performance and reliability.

Key Responsibilities :

Design, develop, and maintain data pipelines using PySpark and related Big Data technologies.

Work with HDFS, Hive, Sqoop, and other tools in the Hadoop ecosystem.

Write efficient HiveQL and SQL queries to handle large-scale datasets.

Perform performance tuning and optimization of distributed data systems.

Collaborate with cross-functional teams in an Agile environment to deliver high-quality solutions.

Manage and schedule workflows using Apache Airflow or Oozie.

Troubleshoot and resolve issues in data pipelines to ensure reliability and accuracy.

Required Skills :

Proven experience in Big Data Engineering with a focus on PySpark.

Strong knowledge of HDFS, Hive, Sqoop, and related tools.

Proficiency in SQL / HiveQL for large datasets.

Expertise in performance tuning and optimization of distributed systems.

Familiarity with Agile methodology and collaborative team practices.

Experience with workflow orchestration tools (Airflow / Oozie).

Strong problem-solving, analytical, and communication skills.

Good to Have :

Knowledge of data modeling and data warehousing concepts.

Exposure to DevOps practices and CI / CD pipelines for data engineering.

Experience with other Big Data frameworks such as Spark Streaming or Kafka.

Create a job alert for this search

Data Pipeline Specialist • Pune, Republic Of India, IN

Related jobs

Promoted

Azure Data Pipeline Specialist

CoforgePune, Republic Of India, IN

The ideal candidate will be responsible for designing, implementing, and maintaining scalable data solutions that support business intelligence and analytics needs. Design, implement, and maintain d...Show moreLast updated: 27 days ago

Promoted

Lead Data Pipeline Engineer

EXLPune, Republic Of India, IN

We are seeking a Senior Data Engineer to build and optimize data systems that power batch processing, real-time streaming, pipeline orchestration, data lake management, and data cataloging.You will...Show moreLast updated: 30+ days ago

Promoted

Data Pipeline Specialist

TEKsystemsRepublic Of India, IN

The position is open with DaaS Core Data Assets Development and Support team, they focus on the serving external customers / clients and the position is open as part of new hiring projection for thei...Show moreLast updated: 25 days ago

Promoted

Azure Data Pipeline Specialist

Tata Consultancy ServicesRepublic Of India, IN

TCS is hiring for Azure Data Engineer.Designing and implementing data storage solutions on Azure.Azure data engineer is to design and implement data storage solutions on Azure.That includes selecti...Show moreLast updated: 2 days ago

Promoted

Data Pipeline Specialist

Persistent SystemsPune, Republic Of India, IN

We are seeking for Data engineer with AWS, databricks experience.Location : All Persistent Location.Job Type : Full Time Employment. Must have knowledge and working experience in Databricks, PySpark a...Show moreLast updated: 2 days ago

Promoted

Data Pipeline Engineer

JMAN GroupChennai, Republic Of India, IN

JMAN Group is a fast-growing data engineering & data science consultancy.We work primarily with Private Equity Funds and their Portfolio Companies to create commercial value using Data & Artificial...Show moreLast updated: 17 days ago

Promoted

Lead Data Pipeline Engineer

Eucloid Data SolutionsChennai, Republic Of India, IN

Eucloid is looking for a Senior Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure for our cli...Show moreLast updated: 14 days ago

Promoted

AI Pipeline Development Specialist

VIVANETPune, Republic Of India, IN

Pune / Hyderabad / Bengaluru - Work from office.With over two decades of experience in technology, leadership, and innovation, we’re now focused on. AI research and frameworks into scalable, production-...Show moreLast updated: 1 day ago

Promoted

Data Pipeline Developer

sliceRepublic Of India, IN

We’ve all felt how slow, confusing, and complicated banking can be.We’re building every product from scratch to be fast, transparent, and feel good, because we believe that the best products transc...Show moreLast updated: 17 days ago

Promoted

GCP Data Pipeline Specialist

HCLTechChennai, Republic Of India, IN

HCL Tech - GCP Data Engineer – Hackathon Opportunity (Face-to-Face, Chennai).HCLTech is inviting experienced and passionate. This event is designed to identify exceptional talent who can architect, ...Show moreLast updated: 17 days ago

Promoted

Azure Data Pipeline Specialist

SystemBenderRepublic Of India, IN

Responsible for designing and maintaining scalable data pipelines on Microsoft Fabric and Azure.Focus includes ingesting structured, semi-structured, and unstructured data, managing OneLake / Delta L...Show moreLast updated: 3 days ago

Promoted

Data Pipeline Engineer

EXLPune, Republic Of India, IN

DWH / ETL Developer : Strong experience in ETL development and data warehousing (SQL, Informatica).Responsibilities for Candidates. Design, develop, and maintain ETL workflows ( Informatica ).Optimize ...Show moreLast updated: 30+ days ago

Promoted

Cloud Data Pipeline Specialist

Capgemini InventPune, Republic Of India, IN

At Capgemini Invent, we believe difference drives change.As inventive transformation consultants, we blend our strategic, creative and scientific capabilities, collaborating closely with clients to...Show moreLast updated: 1 day ago

Promoted

Drilling Data Specialist

SLBRepublic Of India, IN

SLB is the world's leading provider of technology for reservoir characterization, drilling, production, and processing in the oil and gas industry. Working in more than 85 countries and employing ap...Show moreLast updated: 10 days ago

Promoted

Lead Data Pipeline Engineer

MakeMyTripRepublic Of India, IN

At MakeMyTrip (MMT), technology is at the heart of everything we do.As a leading player in the travel industry, we leverage cutting-edge solutions like AI, machine learning, and cloud infrastructur...Show moreLast updated: 17 days ago

Promoted

BigQuery Data Pipeline Specialist

AdastraRepublic Of India, IN

We are looking for a proactive and solution-oriented GCP Data Engineer to join our team.This role requires hands-on experience in Google Cloud Platform (GCP), especially with BigQuery and Airflow, ...Show moreLast updated: 3 days ago

Promoted

Data Pipeline Engineer

YubiRepublic Of India, IN

As a Data Engineer, you will be part of a highly talented Data Engineering team.Responsible for developing reusable capabilities and tools to automate various types of data processing pipelines.You...Show moreLast updated: 30+ days ago

Promoted

Spark Data Pipeline Specialist

HCLTechChennai, Republic Of India, IN

Position : Senior Data Engineer – Experience : 6–9 Years.We are seeking Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kuberne...Show moreLast updated: 4 days ago