Talent.com
Lead Data Pipeline Engineer

Lead Data Pipeline Engineer

EXLHyderabad, Republic Of India, IN
30+ days ago
Job description

Job Summary :

We are seeking an experienced Data Engineer with strong expertise in AWS cloud services and Databricks to design and implement scalable, high-performance data pipelines. The role involves building and optimizing ETL / ELT workflows, managing big data environments, and ensuring seamless data availability for analytics, AI / ML, and business intelligence use cases.

Key Responsibilities :

  • Design, develop, and maintain data pipelines and workflows using AWS (S3, Redshift, Glue, EMR, Lambda, Kinesis, Athena) and Databricks .
  • Build and optimize ETL / ELT processes for large-scale structured and unstructured datasets.
  • Leverage Databricks for data engineering, advanced analytics, and machine learning workloads.
  • Collaborate with data scientists, analysts, and stakeholders to deliver reliable and scalable data solutions.
  • Implement data governance, quality checks, and monitoring frameworks .
  • Ensure high availability, scalability, and cost-effectiveness of data solutions.

Required Skills :

  • 7+ years of experience in data engineering .
  • Strong hands-on experience with AWS cloud services (S3, Redshift, Glue, EMR, Athena, Lambda, Kinesis).
  • Expertise in Databricks (PySpark, Delta Lake, ML pipelines).
  • Proficiency in Python and SQL for data transformation and querying.
  • Strong understanding of data warehousing, data modeling, and big data frameworks .
  • Experience with workflow orchestration tools (Airflow, Step Functions).
  • Preferred Qualifications :

  • Familiarity with MLOps / ML integration within Databricks.
  • Experience with real-time / streaming data (Kafka, Spark Streaming).
  • Knowledge of infrastructure as code (Terraform / CloudFormation).
  • Exposure to data security, compliance, and governance practices.
  • Create a job alert for this search

    Data Pipeline Engineer • Hyderabad, Republic Of India, IN

    Related jobs
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Searce IncHyderabad, IN
    Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 27 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Saras AnalyticsHyderabad, Telangana, India
    We are an ecommerce focused end to end data analytics firm assisting enterprises & brands in data driven decision making to maximize business value. Our suite of work spans extraction, transformatio...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer- Snowflake

    Data Engineer- Snowflake

    Prudent Technologies and Consulting, Inc.Hyderabad, Telangana, India
    Job Description : We are seeking a skilled Data Engineer with strong experience in Python, Snowflake, and AWS.The ideal candidate will be responsible for building and optimizing scalable data pip...Show moreLast updated: 30+ days ago
    • Promoted
    Data Pipeline Engineer

    Data Pipeline Engineer

    ValueLabsHyderabad, Republic Of India, IN
    We are seeking a talented Data Engineer to join our dynamic team at ValueLabs.As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that sup...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer - ETL / Python

    Lead Data Engineer - ETL / Python

    Lera Technologies Pvt. LtdHyderabad
    What You Bring : - 7+ years of experience in Data Engineering, including SQL / PLSQL / TSQL, ETL development, and data pipeline architecture - Strong...Show moreLast updated: 27 days ago
    • Promoted
    Big Data Engineer - Python / PySpark

    Big Data Engineer - Python / PySpark

    People Prime World WideHyderabad
    Key Responsibilities : - Design, develop, and maintain scalable ETL / ELT pipelines using Python and PySpark.Work with large-scale datasets across di...Show moreLast updated: 30+ days ago
    • Promoted
    Backend and Data Pipeline Engineer

    Backend and Data Pipeline Engineer

    JRD Systemshyderabad, India
    Job Role : Backend and Data Pipeline Engineer - Python.We’re investing in technology to develop new products that help our customers drive their growth and transformation agenda.These include new da...Show moreLast updated: 10 days ago
    • Promoted
    Data Engineer – Databricks Platform

    Data Engineer – Databricks Platform

    Amicon Hub ServicesHyderabad, India
    Proven experience in Databricks (Delta Lake, Spark, PySpark, SQL).Strong expertise in data integration from multiple sources (SQL Server, MongoDB, InfluxDB). Hands-on experience with ETL / ELT pipelin...Show moreLast updated: 5 days ago
    • Promoted
    Data Analytics Engineer - ETL

    Data Analytics Engineer - ETL

    Prudential Manpower GroupHyderabad
    Mode of Work : : 5-10 : : 2 PM - 11 PM Job Summary : Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    INFEC Services Private LimitedHyderabad
    About the Role : We are seeking a highly skilled Data Engineer to design, develop, and optimize scalable data pipelines and solutions on Google Cloud Platform (GCP).T...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer Gcp

    Lead Data Engineer Gcp

    EgenHyderabad, Republic Of India, IN
    We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team.The ideal candidate will...Show moreLast updated: 5 days ago
    • Promoted
    GCP Data Engineer

    GCP Data Engineer

    Tata Consultancy ServicesGreater Hyderabad Area, India
    Required Technical Skill Set -.GCP, Python, SQL, Big Query, ETL Data Pipeline.Chennai, Kochi, Bangalore,HYD,Pune.GCP services (Big Query, Cloud Run, Pub / Sub, Cloud Storage, Spanner, Cloud Composer,...Show moreLast updated: 17 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Response InformaticsHyderabad, Telangana, India
    AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture. CICD - Terraform or CloudFo...Show moreLast updated: 15 days ago
    • Promoted
    BPK Tech - Lead Data Engineer - Python

    BPK Tech - Lead Data Engineer - Python

    M / s Bpk Tech Services India Pvt LtdHyderabad
    Position : Lead Data Engineer Experience range 10+ years Primary Skills / Mandatory Skills : Snowflake, Pyt...Show moreLast updated: 30+ days ago
    • Promoted
    ETL Data Engineer

    ETL Data Engineer

    BPMLinksHyderabad, India
    In the digital age, harnessing the power of data is crucial for business success.We offer a holistic approach that encompasses various aspects of data strategy, roadmap development, and data analyt...Show moreLast updated: 5 days ago
    • Promoted
    Data Engineer

    Data Engineer

    NPG ConsultantsHyderabad
    Data Engineer We are looking for a skilled Data Engineer to join our growing team.You will design and maintain scalable data pipelines, integrate cloud platforms, and...Show moreLast updated: 30+ days ago
    • Promoted
    Tech Lead - Data Engineer

    Tech Lead - Data Engineer

    RecroHyderabad, Telangana, India
    Data Engineering and Analytics team.The ideal candidate will be responsible for configuring, deploying, and maintaining Quantexa software on leading cloud platforms while leveraging their expertise...Show moreLast updated: 8 days ago
    • Promoted
    Egen - Lead Data Engineer - Python & GCP

    Egen - Lead Data Engineer - Python & GCP

    SPRINGML INDIA DEVELOPMENT CENTER PRIVATE LIMITEDHyderabad
    Job title : Lead Data Engineer - Python & : : 10- 12 Job Overview : < / b&g...Show moreLast updated: 30+ days ago