Talent.com
Data Engineer - Python/Spark
Data Engineer - Python/SparkO2F Info Solutions Pvt. Ltd. • Chennai
Data Engineer - Python / Spark

Data Engineer - Python / Spark

O2F Info Solutions Pvt. Ltd. • Chennai
30+ days ago
Job description

Job Summary :

We are seeking a highly skilled Senior Data Engineer with 4 to 8 years of experience in building robust data pipelines and working extensively with PySpark to join our data engineering team.

Key Responsibilities :

Data Pipeline Development :

  • Design, build, and maintain scalable data pipelines using PySpark to process large datasets and support data-driven applications and analytics.

ETL Process Automation :

  • Develop and automate ETL (Extract, Transform, Load) processes using PySpark, ensuring efficient data processing, transformation, and loading from diverse sources into data lakes, warehouses, or databases.
  • Distributed Computing with PySpark :

  • Leverage Apache Spark and PySpark to process large-scale data in a distributed computing environment, optimizing for performance and scalability.
  • Cloud Data Solutions :

  • Develop and deploy data pipelines and processing frameworks on cloud platforms (AWS, Azure, GCP) using native tools like AWS Glue, Azure Databricks, or Google Dataproc.
  • Data Integration & Transformation :

  • Integrate data from various internal and external sources, ensuring data consistency, quality, and reliability throughout the pipeline.
  • Performance Optimization :

  • Optimize PySpark jobs and pipelines for faster data processing, handling large volumes of data efficiently with minimal latency.
  • Proven experience as a Data Engineer or similar role, with a strong background in database development, ETL processes, and software development.
  • Proficiency in SQL and scripting languages such as Python, with experience working with relational databases.
  • Proficiency in dataProc (PySpark), Pandas or other data processing libraries
  • Experience with data modeling, schema design, and optimization techniques for scalability.
  • Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize data processing pipelines for scale
  • Required Qualifications :

  • 4-8 years of experience in data engineering, with a strong focus on PySpark and large-scale data processing.
  • Technical Skills :

  • Expertise in PySpark for distributed data processing, data transformation, and job optimization.
  • Strong proficiency in Python and SQL for data manipulation and pipeline creation.
  • Hands-on experience with Apache Spark and its ecosystem, including Spark SQL, Spark Streaming, and PySpark MLlib.
  • Solid experience working with ETL tools and frameworks, such as Apache Airflow or similar orchestration tools.
  • (ref : hirist.tech)

    Create a job alert for this search

    Data Engineer • Chennai

    Related jobs
    Senior Data Engineer - Python / Spark

    Senior Data Engineer - Python / Spark

    Getinz • Chennai
    Description : Role Overview : We are looking for a highly skilled and motivated Senior Data Engineer to join our clients tea...Show more
    Last updated: 27 days ago • Promoted
    Data Engineer-Airflow, Python,Pyspark, Databricks

    Data Engineer-Airflow, Python,Pyspark, Databricks

    NielsenIQ • Chennai, Tamil Nadu, India
    Were seeking a highly motivated Data Engineer to join our agile cross-functional team and drive end-to-end data pipeline development in a cloud-native big data ecosystem. Youll leverage ETL / ELT best...Show more
    Last updated: 30+ days ago • Promoted
    PySpark Data Engineer

    PySpark Data Engineer

    EXTRAGIG • Chennai, IN
    Contract Assistant – Data Engineer Support (Remote, EST Hours).PySpark Data Engineer with daily activities.This is a remote contract role. Execute creative software and data solutions, including des...Show more
    Last updated: 30+ days ago • Promoted
    Senior Python Data Engineer

    Senior Python Data Engineer

    iVoyant • chennai, tamil nadu, in
    Join a dynamic engineering team working on a high-impact tax reporting platform for the 2025 fiscal season.The core goal is to modernize and significantly accelerate the generation of Excel-based r...Show more
    Last updated: 8 days ago • Promoted
    Data Engineer

    Data Engineer

    IntraEdge • chennai, tamil nadu, in
    Python, PySpark, AWS services (Glue, Lambda), and Snowflake.The ideal candidate will design, build, and maintain scalable data pipelines, ensure efficient data integration, and enable advanced anal...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer - Scala Spark

    Data Engineer - Scala Spark

    NielsenIQ • Chennai, India
    Design, build, and optimize large-scale ETL and data-processing pipelines handling GB–TB volumes.Operate within the Databricks ecosystem and drive migration of selected workloads to high-performanc...Show more
    Last updated: 9 days ago • Promoted
    Senior Python Data Engineer

    Senior Python Data Engineer

    SIRO • chennai, tamil nadu, in
    Good Python Programming with 6 years of experience; PyCharm; Well versed in AWS tools , will be good if AWS architect certified. GitHub; GitAction; Experience in deploying Data Pipelines.Good commu...Show more
    Last updated: 13 days ago • Promoted
    Senior Python Data Engineer – ETL & Pipeline Development (5 to 12 yrs)

    Senior Python Data Engineer – ETL & Pipeline Development (5 to 12 yrs)

    AIMLEAP • Chennai, IN
    Senior Python Data Engineer – ETL & Pipeline Development.Remote (WFH) / Bangalore / India.Tech / MCA / Computer Science / IT. IT / Data / AI / LegalTech / Enterprise Solutions.ETL pipelines and data...Show more
    Last updated: 2 days ago • Promoted
    Data Engineer

    Data Engineer

    EXL • Chennai, India
    Collaborate with project stakeholders (client) to identify product and technical requirements.Develop, implement, and tune large-scale distributed systems and pipelines that process large volume of...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer

    Data Engineer

    Vriba Solutions • Chennai, IN
    Design, develop & maintain ETL / ELT pipelines.Ingest & transform data from APIs, DBs, files, streams.Build real-time & batch processing solutions. Data validation, quality & cleansing.Translate busin...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer

    Data Engineer

    Aceolution • Chennai, IN
    Data Engineer – Python Expert(Freelance Role).We are looking for a seasoned Senior Data Engineer to architect, build, and own the data pipelines that power our large language model (LLM) developmen...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer (DOMO & Python / Kafka)

    Data Engineer (DOMO & Python / Kafka)

    Smart Moves Consultants • chennai, tamil nadu, in
    We are looking for candidates with strong Domo and Python / Kafka connect experience.Please find the JD below and share suitable and strong profiles at the earliest. Design, build, and maintain scala...Show more
    Last updated: 17 hours ago • Promoted • New!
    Python Data Engineer

    Python Data Engineer

    Dexian India • Chennai, Tamil Nadu, India
    Design, build, and deploy robust ETL / ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Data...Show more
    Last updated: 21 days ago • Promoted
    Data Engineer (Spark / Scala)

    Data Engineer (Spark / Scala)

    Tata Consultancy Services • Chennai, Tamil Nadu, India
    Greetings from Tata Consulting Services.TCS is Hiring for Data Engineer (Spark / Scala).Required Technical Skill - Scala / Spark , Hadoop, Hive.Show more
    Last updated: 22 days ago • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Straive • chennai, tamil nadu, in
    The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show more
    Last updated: 30+ days ago • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Intellias • chennai, tamil nadu, in
    Apache Flink / Apache Spark (Streaming).Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines. Kafka + Flink / Spark Streaming).Python for data engin...Show more
    Last updated: 21 days ago • Promoted
    Azure Data Engineer - ETL

    Azure Data Engineer - ETL

    Qrata Consulting • Chennai
    Position : Azure Data Engineer Experience : 3 to 5 Years Job Type : Full-time ...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer (Python, SQL, Spark, ETL)

    Data Engineer (Python, SQL, Spark, ETL)

    Confidential • Chennai, India
    Minimum Years of Experience 3-5 years in Python and SQL programming, ETL 2+ years in Spark, data visualization Duties and Responsibilities : Data processing : data querying, analysis and validation B...Show more
    Last updated: 7 days ago • Promoted