Talent.com
Pyspark Data Engineer (Immediate Joiner)

Pyspark Data Engineer (Immediate Joiner)

ConfidentialChennai, India
10 days ago
Job description

We are looking for a skilled Data Engineer with hands-on experience in PySpark and Databricks to design, develop, and optimize large-scale data pipelines. The ideal candidate should have strong problem-solving skills, experience working in cloud-based environments, and the ability to collaborate effectively with cross-functional teams.

Key Responsibilities :

  • Develop and maintain ETL / ELT pipelines using PySpark on Databricks.
  • Optimize data processing workflows for performance and scalability .
  • Work with structured, semi-structured, and unstructured data from multiple sources.
  • Collaborate with data scientists, analysts, and business teams to deliver clean, reliable datasets .
  • Implement data quality checks, validations, and monitoring .
  • Support migration of existing pipelines to cloud platforms (Azure / AWS / GCP).
  • Document technical solutions, workflows, and best practices.

Required Skills :

  • 3–5 years of experience in data engineering roles.
  • Strong expertise in PySpark (DataFrames, RDDs, Spark SQL).
  • Hands-on experience with Databricks (Delta Lake, Workflows, Jobs, Clusters).
  • Solid understanding of data warehousing concepts and SQL .
  • Experience with cloud platforms (Azure Data Lake, AWS S3, or GCP BigQuery).
  • Familiarity with CI / CD pipelines and version control (Git).
  • Good understanding of performance tuning and optimization in Spark.
  • Strong problem-solving and debugging skills.
  • Skills Required

    Spark SQL, Performance Tuning, Pyspark, Optimization, Sql, ELT, Git, Gcp, Databricks, Azure, Aws, Etl

    Create a job alert for this search

    Data Engineer • Chennai, India

    Related jobs
    • Promoted
    • New!
    Backend API Integration Engineer | Immediate Joiner |

    Backend API Integration Engineer | Immediate Joiner |

    Zimetricschennai, tamil nadu, in
    The role involves designing, building, and maintaining scalable backend services and API integrations across enterprise systems and cloud platforms, using. JWT, SAML, and SSO integrations.MuleSoft, ...Show moreLast updated: 12 hours ago
    • Promoted
    • New!
    Need Immediate Joiner (No Notice Period) at Chennai Location Only- Data Analyst with Databricks and Spark / PySpark exp. @ Chennai

    Need Immediate Joiner (No Notice Period) at Chennai Location Only- Data Analyst with Databricks and Spark / PySpark exp. @ Chennai

    E-SolutionsChennai, Tamil Nadu, India
    Title : Data Analyst with Databricks and Spark / PySpark exp.Banking data analytics exposure (risk / finance).SQL + Databricks + Azure stack proficiency. Data mapping, data modeling, and visualization (...Show moreLast updated: 10 hours ago
    • Promoted
    ETL Data Engineer + SQL - (Immediate joiners only)

    ETL Data Engineer + SQL - (Immediate joiners only)

    Innovya TechnologiesChennai, IN
    Innovya Technologies is a dynamic and growing software consulting firm that drives business automation with cutting-edge solutions. We help businesses quickly realize value from their technology and...Show moreLast updated: 1 day ago
    • Promoted
    Data Engineer

    Data Engineer

    Tata Consultancy Serviceschennai, tamil nadu, in
    Skill- Azure Senior Data Engineers.Requires skills-Azure Data Platform - Databricks,AzureSQL and ADLS.Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    VAANTECHchennai, tamil nadu, in
    Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning. Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    VeraxionChennai, IN
    Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions. Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 17 days ago
    • Promoted
    Databricks Data Engineer

    Databricks Data Engineer

    Tata Consultancy ServicesChennai, Tamil Nadu, India
    The Databricks Data Engineer will play a crucial role in migrating and optimizing data workflows, ensuring high performance and reliability in our data operations. Location - Kolkata / Chennai / Trivan...Show moreLast updated: 28 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Response Informaticschennai, tamil nadu, in
    AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture. CICD - Terraform or CloudFo...Show moreLast updated: 30+ days ago
    • Promoted
    GCP Data Engineer

    GCP Data Engineer

    Tata Consultancy Serviceschennai, tamil nadu, in
    Notice Period- 0 to 15 days / Serving Notice period.Skills : GCP, PySpark, Python, HDFS, Hadoop, SQL.Good Hands on knowledge on GCP. Should have worked on Data Migration projects, from On-prem to Clo...Show moreLast updated: 30+ days ago
    • Promoted
    AWS Data Engineer

    AWS Data Engineer

    Tata Consultancy Serviceschennai, tamil nadu, in
    Aws data engineer having strong experience of Python.Technical / Behavioral Competency.Proficient in Python, with experience in deploying Python packages and OOP, Experience in ingesting data from di...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer – ETL & Pipeline Development (5 to 10 yrs)

    Data Engineer – ETL & Pipeline Development (5 to 10 yrs)

    AIMLEAPchennai, tamil nadu, in
    Data Engineer – ETL & Pipeline Development.Remote (Work from Home) / Bangalore / India.Tech / MCA / Computer Science / IT. IT / Data / AI / LegalTech / Enterprise Solutions.ETL workflows — not just ...Show moreLast updated: 1 day ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    IntelliasChennai, IN
    Apache Flink / Apache Spark (Streaming).Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines. Kafka + Flink / Spark Streaming).Python for data engin...Show moreLast updated: 6 days ago
    • Promoted
    Data Engineer

    Data Engineer

    LTIMindtreechennai, tamil nadu, in
    We are hiring for the position of.GCP (Mandatory), BigQuery, SQL.This role is specifically for candidates with.GCP Data Engineering background. Please apply only if you meet the required criteria.In...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Dexian IndiaChennai, Tamil Nadu, India
    Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights. Constructing infrastructure for efficient ETL processes from vari...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer (Spark / Scala)

    Data Engineer (Spark / Scala)

    Tata Consultancy ServicesChennai, Tamil Nadu, India
    Greetings from Tata Consulting Services.TCS is Hiring for Data Engineer (Spark / Scala).Required Technical Skill - Scala / Spark , Hadoop, Hive.Show moreLast updated: 6 days ago
    • Promoted
    Big Data Engineer

    Big Data Engineer

    Tata Consultancy ServicesChennai, Tamil Nadu, India
    Chennai, Bangalore, Hyderabad, Mumbai, Kolkata, Pune, Indore, Ahmedabad, Kochi, Gurgaon.HDFS, Hive, Spark, Sqoop, Flume, Oozie, Unix Script, Autosys. Hands-on experience in Big data, Hadoop.Hands-on...Show moreLast updated: 20 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Straivechennai, tamil nadu, in
    The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    IntraEdgeChennai, IN
    Python, PySpark, AWS services (Glue, Lambda), and Snowflake.The ideal candidate will design, build, and maintain scalable data pipelines, ensure efficient data integration, and enable advanced anal...Show moreLast updated: 30+ days ago