Python, Pyspark

ConfidentialPune

30+ days ago

Job description

Design and implement scalable data processing pipelines using PySpark for batch and real-time data processing.

Develop Python scripts and libraries to integrate, cleanse, and process data from various data sources (databases, APIs, etc.).

Optimize PySpark jobs for performance and resource efficiency.

Collaborate with data engineers and analysts to define data transformation and aggregation requirements.

Analyze and troubleshoot issues in large-scale distributed data systems.

Develop custom functions for data manipulation and aggregation using Spark DataFrame API and RDD API .

Perform data analysis and create insightful visualizations for internal stakeholders.

Write unit tests, document code, and maintain code quality through peer reviews and best practices.

Monitor and optimize the performance of Apache Spark clusters.

Work with cloud platforms such as AWS , Azure , or GCP to deploy Spark jobs and other data processing services.

Skills and Qualifications : Strong experience with Python programming language and libraries such as Pandas , NumPy , Pytest , etc.

Hands-on experience with Apache Spark and PySpark for distributed data processing.

Familiarity with data formats such as CSV , JSON , Parquet , Avro , and other big data file formats.

Solid understanding of Hadoop ecosystem components (HDFS, YARN, etc.).

Proficiency with data wrangling, transformation, and aggregation.

Experience with performance tuning, resource management, and debugging Spark applications.

Understanding of cloud-based data storage solutions (S3, GCS, etc.) and distributed file systems.

Familiarity with data warehousing concepts and tools.

Good knowledge of SQL and relational databases (e.g., PostgreSQL, MySQL).

Strong problem-solving skills and ability to work with large datasets.

Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Preferred Qualifications : Bachelor s or Master s degree in Computer Science, Engineering, or a related field.

Experience with additional big data frameworks like Flink , Kafka , Hive , etc.

Knowledge of Docker and containerization for deployment.

Experience with ETL tools like Airflow or Luigi .

Familiarity with machine learning concepts and frameworks (e.g., MLlib , Scikit-learn ).

Skills Required

Python, Pyspark, Aws, Azure, Csv, Json

Create a job alert for this search

Python Pyspark • Pune

Related jobs

Promoted

Python Developer

Synechronpune, maharashtra, in

About Company : At Synechron, we believe in the power of digital to transform businesses for the better.Our global consulting firm combines creativity and innovative technology to deliver industry-l...Show moreLast updated: 30+ days ago

Promoted

Python Developer

Cavista TechnologiesPune, Maharashtra, India

At Cavista, our mission is to empower organizations with the world’s best technology solutions.We ensure the highest level of client satisfaction through a global network of innovation, and our sof...Show moreLast updated: 30+ days ago

Promoted

Senior Python Developer

Aarav Solutionspune, maharashtra, in

Aarav Solutions is a global Product Engineering and IT Consulting services provider.Our goal is to enable organizations in their digital transformation journey, leveraging our domain knowledge and ...Show moreLast updated: 20 days ago

Promoted

Lead Python Developer_ Exp : 6+ Years

Atyeti IncPune, Maharashtra, India

Job Description : Strong hands-on experience in Python.Strong experience on Django / Flask framework development.Solid understanding of OOP, REST APIs, and microservices architecture.Exposure to SQL...Show moreLast updated: 3 days ago

Promoted

Python Developer

CozzeraPune, IN

Shift : Normal IST Business Hours.We are seeking a Python / FastAPI Development Lead with strong expertise in FastAPI.The role involves designing, building, and optimizing scalable APIs, collaborating...Show moreLast updated: 20 days ago

Promoted

Python Developer

ProDT Consulting Servicespune, maharashtra, in

Job Title : Tech Lead – Python Full Stack Developer (AI & SaaS).Contact Person -Anuja Deshmukh.Tech Lead with strong Python Full Stack expertise. The ideal candidate should have hands-on experience l...Show moreLast updated: 3 days ago

Promoted

Python Developer

People Prime WorldwidePune, IN

Our client is a Palo Alto–based AI infrastructure and talent platform founded in 2018.It helps companies connect with remote software developers using AI-powered vetting and matching technology.Ori...Show moreLast updated: 30+ days ago

Promoted

Python Pyspark SDET

InfoBeanspune, maharashtra, in

SDET – PySpark / Python Data Engineering (Azure Cloud).Indore | Pune | Bangalore | Chennai.Seeking a skilled Software Development Engineer in Test (SDET) with expertise in PySpark and Python for data...Show moreLast updated: 1 day ago

Promoted

Python Developer

CodeVyasaPune, IN

We are seeking a highly skilled and motivated Python Developer|| Remote with.If you're passionate about coding, problem-solving, and. CodeVyasa is a mid-sized product engineering company that works ...Show moreLast updated: 22 days ago

Promoted

Senior Python Developer

Huptech HR SolutionsPune, IN

Job Post : - Senior Python Developer.Software Development : Write clean, scalable, and efficient Python code to develop software applications. Backend Development : Design, develop, and maintain server...Show moreLast updated: 3 days ago

Promoted

Senior Python Developer

Whitefield CareersPune, Maharashtra, India

Qualifications - Bachelor’s / Master’s degree in Computer Science, Engineering, or related field.Excellent communication and leadership skills are a must for this role. Effectively conveying ideas an...Show moreLast updated: 1 day ago

Promoted
New!

Python Developer

EC Infosolutions Pvt LtdPune, Maharashtra, India

Job description Job Title : Python.EC Infosolutions is a forward-thinking company with a focus on technology solutions.We pride ourselves on our commitment to innovation, excellence, and customer sa...Show moreLast updated: 14 hours ago

Promoted

Senior Python Developer

IntelliasPune, IN

We seek a Senior Python Software Engineer to join our Recommendations team and bring new features to life, and support our products' move to modern architecture. You will work closely with the produ...Show moreLast updated: 3 days ago

Promoted

Senior Python Developer

Intellify SolutionsPune, IN

We are looking for an experienced.The ideal candidate should have a solid understanding of modern backend systems, architecture design, and performance optimization. The role involves working on sca...Show moreLast updated: 1 day ago

Promoted
New!

Python Developer

NEILSOFT LIMITEDPune, Maharashtra, India

Neilsoft is a 1100+ people engineering services & solutions company addressing international clients in the AEC (Buildings / Infrastructure), Plant engineering, Energy / Water, Industrial machinery...Show moreLast updated: 12 hours ago

Promoted

AWS Python

Tata Consultancy Servicespune, maharashtra, in

Job Title : Senior Python Developer (AWS).We are looking for an experienced Python Developer with strong AWS expertise to join our growing engineering team. The ideal candidate will have hands-on exp...Show moreLast updated: 3 days ago

Promoted

Python Developer

Associativepune, maharashtra, in

Welcome to Associative, a premier software development and consulting powerhouse headquartered in the vibrant technological hub of Pune, Maharashtra, India. Founded as a dynamic partnership firm on ...Show moreLast updated: 2 days ago

Promoted

Python Developer

IntelliasPune, IN

Urgent requirement (serving or 15 days notice period).Experience with Distributed Applications.Experience in working with Machine Learning models. Good understanding of data structures and clean cod...Show moreLast updated: 18 days ago