Talent.com
This job offer is not available in your country.
Freelance Data Engineer

Freelance Data Engineer

upGradTiruppur, IN
3 days ago
Job description

We are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a modern, cloud-native data stack. You'll play a crucial role in transforming raw data into actionable insights, ensuring data quality, and maintaining the infrastructure required for seamless data flow.

Key Responsibilities

  • Develop, construct, test, and maintain robust and scalable large-scale ETL pipelines using PySpark for processing and Apache Airflow for workflow orchestration.
  • Design and implement both Batch ETL and Streaming ETL processes to handle various data ingestion requirements.
  • Build and optimize data structures and schemas in cloud data warehouses like AWS Redshift .
  • Work extensively with AWS data services, including AWS EMR for big data processing, AWS Glue for serverless ETL, and Amazon S3 for data storage.
  • Implement and manage real-time data ingestion pipelines using technologies like Kafka and Debezium for Change Data Capture (CDC).
  • Interact with and integrate data from various relational and NoSQL databases such as MySQL , PgSQL (PostgreSQL) , and MongoDB .
  • Monitor, troubleshoot, and optimize data pipeline performance and reliability.
  • Collaborate with data scientists, analysts, and other engineering teams to understand data needs and deliver high-quality, reliable data solutions.
  • Ensure data governance, security, and quality across all data platforms.

Required Skills & Qualifications

Technical Skills

  • Expert proficiency in developing ETL / ELT solutions using PySpark .
  • Strong experience in workflow management and scheduling tools, specifically Apache Airflow .
  • In-depth knowledge of AWS data services including :
  • AWS EMR (Elastic MapReduce)
  • AWS Glue
  • AWS Redshift
  • Amazon S3
  • Proven experience implementing and managing data streams using Kafka .
  • Familiarity with Change Data Capture (CDC) tools like Debezium .
  • Hands-on experience with diverse database technologies : MySQL , PgSQL , and MongoDB .
  • Solid understanding of data warehousing concepts, dimensional modeling, and best practices for both batch and real-time data processing.
  • Proficiency in a scripting language, preferably Python .
  • General Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
  • Excellent problem-solving, analytical, and communication skills.
  • Ability to work independently and collaboratively in a fast-paced, dynamic environment.
  • Nice to Have (Preferred Skills)

  • Experience with Infrastructure as Code (e.g., Terraform, CloudFormation).
  • Knowledge of containerization technologies (Docker, Kubernetes).
  • Familiarity with CI / CD pipelines.
  • Create a job alert for this search

    Data Engineer • Tiruppur, IN

    Related jobs
    • Promoted
    Freelance Data Engineer

    Freelance Data Engineer

    upGradCoimbatore, IN
    We are seeking a highly skilled and motivated.The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a ...Show moreLast updated: 3 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Digivance SolutionsErode, IN
    Chennai, Bengaluru, Pune, Hyderabad, Mumbai, Delhi NCR.Collaborate with business and technology stakeholders to understand current and future data requirements. Design, build, and maintain reliable,...Show moreLast updated: 24 days ago
    • Promoted
    Data Engineer

    Data Engineer

    TalogyTiruppur, IN
    This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    NAF Technology IndiaCoimbatore, Tamil Nadu, India
    Do you love a Challenge? Are you Passionate about technology? Are you customer obsessed? Looking to innovate? Join NAF Tech! New American Funding (NAF) is a mortgage lender offering an array of loa...Show moreLast updated: 2 days ago
    • Promoted
    GCP Data Engineer

    GCP Data Engineer

    EXLCoimbatore, IN
    Must Have- GCP Data Engineer with Banking / Finance Institutions Experience.Google Cloud Platform (GCP) Engineers.This solution will integrate data from diverse sources including.SAS systems, Excel ...Show moreLast updated: 4 days ago
    • Promoted
    Data Engineer

    Data Engineer

    EveriseErode, IN
    Join us on our mission to elevate customer experiences for people around the world.As a member of the Everise family, you will be part of a global experience company that believes in being people-f...Show moreLast updated: 26 days ago
    • Promoted
    Data Engineer

    Data Engineer

    IntraEdgeTiruppur, IN
    The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both interna...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Manuh TechnologiesCoimbatore, IN
    S3, Glue, Redshift, EMR, Lambda).Develop automation scripts and tools using.Collaborate with data analysts, data scientists, and business stakeholders to ensure data availability and reliability.Tr...Show moreLast updated: 30+ days ago
    • Promoted
    Data Platform Engineer

    Data Platform Engineer

    Intuitive.CloudTiruppur, IN
    With the reputation of being a.Digital Transformation challenges across following Intuitive Superpowers : .Application & Database Modernization. Platform Engineering (IaC / EaC, DevSecOps & SRE).Cloud N...Show moreLast updated: 25 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Jade Business Services (JBS)Coimbatore, IN
    Data Engineer (AI Knowledge Base Platform).The engineer will be responsible for ensuring reliable, scalable, and secure data movement across multiple sources, powering Retrieval-Augmented Generatio...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer (Big Data / Kafka) - Remote / Contract / NOT for freelancers / Independent Contractors

    Data Engineer (Big Data / Kafka) - Remote / Contract / NOT for freelancers / Independent Contractors

    DautomCoimbatore, IN
    Remote
    Job Description : Data Engineer (Big Data / Kafka).We are seeking a highly experienced Senior Data Engineer with a deep background in Big Data technologies to join our team. This is a contract role for...Show moreLast updated: 3 days ago
    • Promoted
    Data Engineer

    Data Engineer

    KeasisTiruppur, IN
    We are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS / Hive / Impala) into Apache Iceberg tables, with dow...Show moreLast updated: 4 days ago
    • Promoted
    Data Engineer

    Data Engineer

    XebiaTiruppur, IN
    We’re Hiring : Data Engineer | Xebia.Any Xebia location (Hybrid, 3 days office per week).Immediate to 2 weeks – only apply if you can join early. Databricks, Python, SQL, and Postgres.The ideal candi...Show moreLast updated: 25 days ago
    • Promoted
    Data Engineer

    Data Engineer

    AAA GlobalErode, IN
    Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team. In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 4 days ago
    • Promoted
    • New!
    Data Engineer

    Data Engineer

    OWOWCoimbatore, IN
    Data Architecture & Infrastructure (40%) .Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery) . Build scalable data pipelines for real-time conversation proc...Show moreLast updated: 6 hours ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Searce IncTiruppur, IN
    Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 15 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    InfogainCoimbatore, IN
    Big Data Engineer (Lead) : As a Big Data Engineer (Lead), you will be responsible for leading a team of big data engineers. You will work closely with clients and team members to understand their req...Show moreLast updated: 30+ days ago
    • Promoted
    SAP Data Engineer (DataSphere / BW / HANA / Data Modeling)

    SAP Data Engineer (DataSphere / BW / HANA / Data Modeling)

    KPG99 INCTiruppur, IN
    Role : SAP Data Engineer (DataSphere / BW / HANA / Data Modeling).Duration : 12-month contract with extensions.Provide state-of-the-art technical support for the SAP Datasphere, BW4 Systems maintenance and...Show moreLast updated: 2 days ago
    • Promoted
    Cloud Data Engineer

    Cloud Data Engineer

    LemongrassCoimbatore, IN
    Lemongrass is a software-enabled services provider, synonymous with SAP on Cloud, focused on delivering superior, highly automated Managed Services to Enterprise customers.Our customers span multip...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer - Quant Trading

    Data Engineer - Quant Trading

    Deeter Investments LLPTiruppur, IN
    Deeter Investments is a founder‑led proprietary trading firm built around real‑time, data‑driven decision‑making.We prize curiosity, collaboration, and a bias for action. After years of discretionar...Show moreLast updated: 4 days ago