Data Engineer (PySpark + Cloudera)zorba ai • Pune, India

Data Engineer (PySpark + Cloudera)

zorba ai • Pune, India

30+ days ago

Job description

Job Summary

We are looking for experienced Data Engineers with strong expertise in PySpark and Cloudera Data Platform to design, develop, and optimize scalable data pipelines. The ideal candidate should have hands-on experience with distributed data systems, cloud platforms (AWS), and modern data architecture, along with a strong understanding of data governance and cataloging tools.

Key Responsibilities

Design, build, and maintain scalable batch and real-time data pipelines using PySpark
Work with Cloudera Data Platform (CDP) components such as CDE, CDW, Ozone, and Airflow
Manage and optimize data workflows, ensuring high performance and reliability
Implement data governance, security, and access control using Apache Ranger
Develop and maintain data models, Hive Metastore, and large-scale distributed datasets
Collaborate with cross-functional teams to deliver data solutions for analytics and reporting
Work with AWS services like EMR, S3, MWAA, Glue Catalog, and Lake Formation
Ensure proper data partitioning, bucketing, and optimization using formats like Iceberg and Parquet
Integrate data cataloging and lineage using Atlan

Required Skills & Qualifications

6+ years of experience in Data Engineering
Strong hands-on experience with PySpark
Deep understanding of modern data platforms and distributed data systems
Experience with Cloudera Data Platform (CDP) ecosystem
Proficiency in SQL and data modeling concepts
Experience with AWS data services (EMR, S3, MWAA, Glue, Lake Formation)
Strong knowledge of Hive Metastore and big data architectures
Experience with file formats (Iceberg, Parquet) and optimization techniques
Familiarity with data governance, cataloging, and lineage tools (Atlan)

Skills: pyspark,aws,cloudera
Skills Required
Pyspark, Sql, Aws

Create a job alert for this search

Data Engineer (PySpark + Cloudera) • Pune, India

Similar jobs

Sr. Data Engineer

Ness Digital EngineeringPune, India, IN

Contract (6 Months, extendable based on performance and project needs).Bangalore (Hyderabad, Pune, Mumbai).We are looking for a SAS ETL Expert with strong SQL expertise.The role involves building s... Show more

Senior Data Engineer

Parable AssociatesPune, Pune, IN

At Parable Associates, we specialize in transforming complex data into clear, actionable insights for our clients across various industries.We’re not just another business intelligence company; we’... Show more