Data Pipeline SpecialistAtomic North • Pune, Republic Of India, IN

No longer accepting applications

Data Pipeline Specialist

Atomic North • Pune, Republic Of India, IN

12 days ago

Job description

Big Data Engineer

Location : Pune / Nagpur (WFO)

Experience Required : 8+ Years

Long term onsite to Abu Dhabi

Job Summary :

We are looking for an experienced Big Data Engineer with strong expertise in PySpark and Big Data ecosystems.

The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high performance and reliability.

Key Responsibilities :

Design, develop, and maintain data pipelines using PySpark and related Big Data technologies.

Work with HDFS, Hive, Sqoop, and other tools in the Hadoop ecosystem.

Write efficient HiveQL and SQL queries to handle large-scale datasets.

Perform performance tuning and optimization of distributed data systems.

Collaborate with cross-functional teams in an Agile environment to deliver high-quality solutions.

Manage and schedule workflows using Apache Airflow or Oozie.

Troubleshoot and resolve issues in data pipelines to ensure reliability and accuracy.

Required Skills :

Proven experience in Big Data Engineering with a focus on PySpark.

Strong knowledge of HDFS, Hive, Sqoop, and related tools.

Proficiency in SQL / HiveQL for large datasets.

Expertise in performance tuning and optimization of distributed systems.

Familiarity with Agile methodology and collaborative team practices.

Experience with workflow orchestration tools (Airflow / Oozie).

Strong problem-solving, analytical, and communication skills.

Good to Have :

Knowledge of data modeling and data warehousing concepts.

Exposure to DevOps practices and CI / CD pipelines for data engineering.

Experience with other Big Data frameworks such as Spark Streaming or Kafka.

Create a job alert for this search

Data Pipeline Specialist • Pune, Republic Of India, IN

Related jobs

Azure Data Pipeline Specialist

Coforge • Pune, Republic Of India, IN

The ideal candidate will be responsible for designing, implementing, and maintaining scalable data solutions that support business intelligence and analytics needs. Design, implement, and maintain d...Show more

Last updated: 30+ days ago • Promoted

Snowflake Data Pipeline Specialist

Synechron • Republic Of India, IN

We have immediate opportunity for Data Engineer – 3-7 Years.We began life in 2001 as a small, self-funded team of technology specialists. Since then, we’ve grown our organization to 14,500+ people, ...Show more

Last updated: 4 days ago • Promoted

Cloud Data Pipeline Specialist

ACL Digital • Republic Of India, IN

We have urgent openings for AWS Data Engineer.The ideal candidate will have a strong background in designing, deploying, and maintaining data pipelines using industry-leading tools such as.Familiar...Show more

Last updated: 9 days ago • Promoted

Data Pipeline Engineer

Theron Solutions • Chennai, Republic Of India, IN

We are seeking a highly skilled ETL Developer with 5–8 years of experience in data integration, transformation, and pipeline optimization. This role is a key part of our Data Engineering function wi...Show more

Last updated: 20 hours ago • Promoted • New!

Cloud Data Pipeline Specialist

HCLTech • Chennai, Republic Of India, IN

Develop and maintain data models and transformations using DBT.Load and manage data in Snowflake, including staging and curated layers. Implement data ingestion processes from various sources using ...Show more

Last updated: 4 days ago • Promoted

Azure Data Pipeline Specialist

Tata Consultancy Services • Republic Of India, IN

TCS is hiring for Azure Data Engineer.Designing and implementing data storage solutions on Azure.Azure data engineer is to design and implement data storage solutions on Azure.That includes selecti...Show more

Last updated: 7 days ago • Promoted

Data Pipeline Specialist

Persistent Systems • Pune, Republic Of India, IN

We are seeking for Data engineer with AWS, databricks experience.Location : All Persistent Location.Job Type : Full Time Employment. Must have knowledge and working experience in Databricks, PySpark a...Show more

Last updated: 7 days ago • Promoted

Data Pipeline Engineer

Luxoft • Chennai, Republic Of India, IN

This project is part of a strategic initiative to migrate legacy on-premises data systems to a modern, scalable cloud-based data platform using Snowflake on Azure. The goal is to enhance data access...Show more

Last updated: 1 day ago • Promoted

Data Pipeline Engineer

JMAN Group • Chennai, Republic Of India, IN

JMAN Group is a fast-growing data engineering & data science consultancy.We work primarily with Private Equity Funds and their Portfolio Companies to create commercial value using Data & Artificial...Show more

Last updated: 22 days ago • Promoted

GCP Data Pipeline Specialist

HCLTech • Chennai, Republic Of India, IN

HCL Tech - GCP Data Engineer – Hackathon Opportunity (Face-to-Face, Chennai).HCLTech is inviting experienced and passionate. This event is designed to identify exceptional talent who can architect, ...Show more

Last updated: 22 days ago • Promoted

Data Pipeline Engineer

EXL • Pune, Republic Of India, IN

DWH / ETL Developer : Strong experience in ETL development and data warehousing (SQL, Informatica).Responsibilities for Candidates. Design, develop, and maintain ETL workflows ( Informatica ).Optimize ...Show more

Last updated: 30+ days ago • Promoted

Cloud Data Pipeline Specialist

Capgemini Invent • Pune, Republic Of India, IN

At Capgemini Invent, we believe difference drives change.As inventive transformation consultants, we blend our strategic, creative and scientific capabilities, collaborating closely with clients to...Show more

Last updated: 7 days ago • Promoted

Data Pipeline Support Specialist

Kumaran Systems • Republic Of India, IN

We’re looking for a skilled Data Support Engineer with.The ideal candidate will serve as the escalation point for data ingestion, pipeline, and consumption issues while ensuring high system stabili...Show more

Last updated: 1 day ago • Promoted

Data Pipeline Development Specialist

Peoplefy • Pune, Republic Of India, IN

We’re Hiring : IT Application Engineer – Informatica Developer (4–6 Years Experience).IT Application Engineer – Applications (Informatica Developer). If you excel at developing scalable data pipeline...Show more

Last updated: 1 day ago • Promoted

Lead Data Pipeline Engineer

MakeMyTrip • Republic Of India, IN

At MakeMyTrip (MMT), technology is at the heart of everything we do.As a leading player in the travel industry, we leverage cutting-edge solutions like AI, machine learning, and cloud infrastructur...Show more

Last updated: 22 days ago • Promoted

Data Pipeline Automation Specialist

Keyrus • Pune, Republic Of India, IN

We have an exciting opportunity in Data Services team for a ETL, Automation Test Engineer to develop and maintain test, data validation frameworks. The Test Engineer will ensure stability of data pi...Show more

Last updated: 9 days ago • Promoted

Data Pipeline Specialist

Bloom AI • New Delhi, Republic Of India, IN

Bloom AI is a modern intelligence layer that accelerates decision-making through AI-driven synthesized insights.We empower enterprises to unlock the value of data with human-like synthesis and deci...Show more

Last updated: 2 days ago • Promoted

Spark Data Pipeline Specialist

HCLTech • Chennai, Republic Of India, IN

Position : Senior Data Engineer – Experience : 6–9 Years.We are seeking Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kuberne...Show more

Last updated: 10 days ago • Promoted