Talent.com
This job offer is not available in your country.
Data Engineer - Chennai / Bengaluru

Data Engineer - Chennai / Bengaluru

PhotonIndia
17 days ago
Job description

Data Engineer : Job Description :

  • Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
  • Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
  • Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
  • Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
  • Design and implement data CI / CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
  • Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
  • Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
  • Stay up-to-date with emerging technologies and best practices in data engineering.
  • Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Responsibilities :

  • Bachelor's degree in Computer Science, Engineering, or a related field
  • 5 + years experience in data engineering, ELT development, and data modeling.
  • Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
  • Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
  • Demonstrated experience in developing custom connectors for data ingestion from various sources.
  • Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
  • Experience implementing DataOps principles and practices, including data CI / CD pipelines.
  • Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
  • Effective communication and collaboration abilities, with a proven track record of working in cross-functional teams.
  • Familiarity with data visualization tools Apache SuperSet and dashboard development.
  • Understanding of distributed systems and working with large-scale datasets.
  • Familiarity with data governance frameworks and practices.
  • Knowledge of data streaming and real-time data processing technologies (., Apache Kafka).
  • Strong understanding of software development principles and practices, including version control (., Git) and code review processes.
  • Experience with Agile development methodologies and working in cross-functional Agile teams.
  • Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
  • Excellent analytical and problem-solving skills, with a keen attention to detail.
  • Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.
  • Required Skills –

    Python, Pyspark,  Sql ,Airflow, Trino, Hive, Snowflake, Agile Scrum

    Good to have–

    Linux,Openshift, Kubernentes, Superset

    Create a job alert for this search

    Data Engineer • India

    Related jobs
    • Promoted
    AWS Data Engineer - ETL / Amazon Athena

    AWS Data Engineer - ETL / Amazon Athena

    DeqodeMumbai
    Hiring : AWS Data Engineer Experience : 5-7 Years Experience | Hybrid | Bengaluru, Pune, Chennai, Mumbai, Gurugram &...Show moreLast updated: 30+ days ago
    Data Engineer - GTP - Chennai

    Data Engineer - GTP - Chennai

    Getinz Techno ServicesChennai, Tamil Nadu, India
    Experience Range : 5 to 9 .Looking for candidates to join in less than or 30 days notice period.Bachelor’s degree in Computer Science, Information Technology, or a related field.Our client is a...Show moreLast updated: 30+ days ago
    Azure Integration Developer

    Azure Integration Developer

    LTIMindtreeIndia
    Azure Integration Developer Experience - 4 To 12 Years Notice Period - Immediate to 30 days Location - Mumbai.Pune, Hyderabad, Bengaluru, Chennai. Skills - Azure Logic App, Function Apps, Azure Data...Show moreLast updated: 30+ days ago
    Junior Sales Engineer – Chennai & Bengaluru

    Junior Sales Engineer – Chennai & Bengaluru

    Hauff-Technik India Pvt LtdIndia
    Company Name : Hauff Technik India Pvt LTD.Junior Sales Engineer Location : .Job Summary : We are looking for enthusiastic and self-motivated. This is an excellent opportunity for candidates looking to ...Show moreLast updated: 1 day ago
    Senior Data Engineer - Chennai

    Senior Data Engineer - Chennai

    Saaki, Argus & Averil ConsultingChennai, Tamil Nadu
    Senior Data Engineer – Data Products.Are you a hands-on data engineer who thrives on solving complex data challenges and building modern cloud-native solutions? We're looking for.This is an excitin...Show moreLast updated: 17 days ago
    Data Scientist- Chennai

    Data Scientist- Chennai

    InvokhrChennai, Tamil Nadu, India
    Quick Apply
    Collaborate with interdisciplinary scientists, IT, and engineering professionals.Solve critical problems and support business decision-making. Strong communication and organizational skills are esse...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    Solution Architect Bengaluru Chennai

    Solution Architect Bengaluru Chennai

    ValueLabsIN
    Competitive / Paid in Indian Rupee.What You Should Know About This Job.Hope you all are doing safe and well.Hiring Solution Architect Bengaluru Chennai. Experience with building end-end design patte...Show moreLast updated: 8 hours ago
    Java Developer_Bengaluru / Chennai

    Java Developer_Bengaluru / Chennai

    Trigent Software Private LimitedKA, India
    Quick Apply
    Experience 8+ years Detailed JD : Strong proficiency in Java programming language, including core Java concepts...Show moreLast updated: 5 days ago
    Sales Associate - Bengaluru / Hyderabad / Chennai

    Sales Associate - Bengaluru / Hyderabad / Chennai

    LenskartBengaluru
    Sales Associate for our Retail Stores who truly puts our customers at the center of everything they do.In this role, you will play a pivotal part in generating sales for our company while prioritiz...Show moreLast updated: 17 days ago
    GenAI Engineer

    GenAI Engineer

    AscendionIndia
    Bengaluru, Pune, Chennai, Hyderabad and Vadodara Years of Exp : .Details : Strong programming background in Python.Deep understanding of LLMs, vector databases, and semantic retrieval.Familiarity with...Show moreLast updated: 1 day ago
    Hiring For Java Cloud

    Hiring For Java Cloud

    krazy mantra hr solutionsINDIA
    Skills : Java engineer, cloud environment, data and analytics projects, manage data life cycle on cloud and use / design distributed computing options such as Kubernetes architecture.Show moreLast updated: 30+ days ago
    • Promoted
    Quality Assurance Tester

    Quality Assurance Tester

    Innova SolutionsGurugram, Haryana, India
    We're Hiring : QA Engineer (Mid-Senior Level).Gurgaon, Chennai, or Bengaluru (Hybrid).Flexibility to support US hours partially. Are you a passionate QA professional ready to take on exciting challen...Show moreLast updated: 1 day ago
    Jobs For Electrical Engineer-Reputed Oil and Gas Industry-Bangalore, Bengaluru, Karnataka, India-5 Lakhs-Vandana

    Jobs For Electrical Engineer-Reputed Oil and Gas Industry-Bangalore, Bengaluru, Karnataka, India-5 Lakhs-Vandana

    Seven ConsultancyBangalore, Bengaluru, Karnataka, India
    Experience in Electrical Design, detail engineering, preparation of layout drawings, cable routing etc.Experience in cable sizing calculation and selection of electrical equipments.Knowledge in Haz...Show moreLast updated: 30+ days ago
    Data Engineer

    Data Engineer

    Tredence Inc.Pune, Maharashtra, India
    Experience Level : 12 months to 16 months only (Excluding internship).Job location- Bengaluru, Chennai, Gurgaon, Kolkata, Pune, Hyderabad. Minimum of 12 months of Data Engineering experience.Strong t...Show moreLast updated: 10 days ago
    Senior Data Engineer - Chennai

    Senior Data Engineer - Chennai

    Saaki Argus & Averil ConsultingChennai, Tamil Nadu, India
    Quick Apply
    Senior Data Engineer Data Products.Are you a hands-on data engineer who thrives on solving complex data challenges and building modern cloud-native solutions? We're looking for.This is an excitin...Show moreLast updated: 30+ days ago
    Data engineer - Chennai

    Data engineer - Chennai

    CITIINDIA
    We are looking for a hands-on Data Engineer who is passionate about solving business problems through innovation and engineering practices. As a Data Engineer, the candidate will leverage deep techn...Show moreLast updated: 30+ days ago
    Test Data Manager

    Test Data Manager

    People Prime WorldwideHyderabad, Telangana, India
    About Company : Our client is prominent Indian multinational corporation specializing in information technology (IT), consulting, and business process services and its headquartered in Bengaluru wi...Show moreLast updated: 1 day ago
    Lead Gen AI Engineer - Immediate Joiners (Bengaluru, Chennai, Gurgaon & Pune)

    Lead Gen AI Engineer - Immediate Joiners (Bengaluru, Chennai, Gurgaon & Pune)

    XebiaGurugram, Haryana, India
    Xebia is a trusted advisor in the modern era of digital transformation, serving hundreds of leading brands worldwide with end-to-end IT solutions. The company has experts specializing in t.AI, digit...Show moreLast updated: 13 days ago