Talent.com
Lead Data Engineer

Lead Data Engineer

Delta Air LinesBengaluru, India
30+ days ago
Job description

Responsibilities :

Data Pipeline Development and Maintenance :

  • Design, build, and optimize scalable ETL / ELT pipelines using AWS Serverless services (Lambda, Glue, s3, Airflow) to ingest data from diverse sources such as APIs, cloud platforms, and databases.
  • Ensure pipelines are robust, efficient, and capable of handling large volumes of data.

Data Integration and Harmonization :

  • Aggregate and normalize marketing performance data from multiple platforms (e.g., Adobe Analytics, Salesforce Marketing Cloud, Teradata data warehouse, ad platforms).
  • Implement data transformation and enrichment processes to support analytics and reporting needs.
  • Data Quality and Monitoring :

  • Develop and implement data validation and monitoring frameworks to ensure data accuracy and consistency.
  • Troubleshoot and resolve issues related to data quality, latency, or performance.
  • Collaboration with Stakeholders :

  • Partner with marketing teams, analysts, and data scientists to understand data requirements and translate them into technical solutions.
  • Provide technical support and guidance on data-related issues or projects.
  • Tooling and Automation :

  • Leverage cloud-based solutions and frameworks (e.g., AWS serverless services - Lambda, Glue, s3, Airflow) to streamline processes and enhance automation.
  • Maintain and optimize existing workflows while continuously identifying opportunities for improvement.
  • Documentation and Best Practices :

  • Document pipeline architecture, data workflows, and processes for both technical and non-technical audiences.
  • Follow industry best practices for version control, security, and data governance.
  • Continuous Learning and Innovation :

  • Stay current with industry trends, tools, and technologies in data engineering and marketing analytics.
  • Recommend and implement innovative solutions to improve the scalability and efficiency of data systems.
  • Lead and mentor other junior members within the team
  • What you need to succeed (minimum qualifications) :

  • Bachelor of Science degree in Computer Science or equivalent
  • At least 6+ years of post-degree professional experience as a data engineer developing and maintaining data pipelines
  • Extensive experience with databases and data platforms (Teradata and AWS serverless - (Lambda, Glue, s3, Airflow)
  • 5+ Hands-on experience in designing, implementing, managing large scale data and ETL solutions utilizing AWS Compute, Storage and database services (S3, Lambda, Redshift, Glue, Athena)
  • 3+ years of experience as a tech lead with experience in mentorship and guidance, task planning and delegation, collaboration across teams, and technical communication
  • Should have experience in architecture and design decisions, code quality and reviews, hands-on development and tech stack and tools management
  • Should have good working knowledge on sprint planning and execution, risk management, performance and stability focus
  • Proficiency in Python, SQL, PySpark, Glue, Lambda, S3 and other AWS tool sets
  • Good understanding of data warehouses, ETL, AWS serverless
  • Strong analytical and programming skills with the ability to solve data-related challenges efficiently
  • Collaborate with product owners to understand requirements and translate them into technical specs
  • Perform regular code reviews and advocate for best practices (testing, CI / CD, documentation)
  • Proven ability to learn new data models quickly and apply them effectively in a fast-paced environment.
  • Excellent communication skills with the ability to present complex data findings to both technical and non-technical audiences.
  • What will give you a competitive edge (preferred qualifications) :

  • Desired Airline industry experience
  • Desired experience working with marketing and media data
  • Experience working with SAS (Statistical Analysis System) to develop data pipelines
  • AWS certifications : Solution Architect (SAA / SAP) or Data Analytics Specialty (DAS)
  • Experience migrating data pipelines and systems to modern cloud-based solutions
  • Familiarity with marketing data platforms
  • ID : DELYB03

    Create a job alert for this search

    Lead Data Engineer • Bengaluru, India

    Related jobs
    • Promoted
    Lead Data Engineer – Snowflake & dbt

    Lead Data Engineer – Snowflake & dbt

    LumenDataBengaluru, Karnataka, India
    Lead Data Engineer – Snowflake & dbt.LumenData is a leading consulting firm specializing in enterprise data management, analytics, and cloud modernization. We partner with clients across industries ...Show moreLast updated: 17 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    CoffeeBeansBengaluru, Karnataka, India
    Founded in the year 2017, CoffeeBeans specialises in offering high end consulting services in technology, product, and processes. We help our clients attain significant improvement in quality of del...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer [T500-20510]

    Lead Data Engineer [T500-20510]

    ANSRBengaluru, Karnataka, India
    ANSR is hiring for one of its clients.Martens is an iconic British brand founded in 1960 in Northamptonshire.Produced originally for workers looking for tough, durable boots, the brand was quickly ...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Alp Consulting Ltd.Bengaluru, Karnataka, India
    Please find the below JD for your reference : .Hands-on experience with AWS or similar cloud platform building data engineering solutions for analytics and science. Must have experience building compl...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Tata Consultancy ServicesBengaluru, India
    Desired Experience Range : 10 - 17 years.Location of Requirement : Bangalore.Minimum of 8-10 years of relevant data science, Business Intelligence, or data analytics experience – preferably with a di...Show moreLast updated: 17 days ago
    • Promoted
    GCP Data Architect

    GCP Data Architect

    Tata Consultancy Serviceshosur, tamil nadu, in
    Job Title : - GCP Data Architect.Design and Implement Data Architectures : Architect and build scalable, end-to-end data solutions on GCP, encompassing data ingestion, transformation, storage, and co...Show moreLast updated: 17 days ago
    • Promoted
    Lead Azure Engineer

    Lead Azure Engineer

    Its All About PeopleBangalore
    Job Description : Key Responsibilities : - Lead the d...Show moreLast updated: 30+ days ago
    • Promoted
    Technical Lead-Data Engineer

    Technical Lead-Data Engineer

    Best Infosystems Ltd.Bengaluru, Karnataka, India
    Technical Lead-Data Engineer_Full-Time_Bangalore / Pune / Navi Mumbai / Noida / Hyderabad / Chennai.Bangalore / Pune / Navi Mumbai / Noida / Hyderabad / Chennai. Bachelor’s / relevant exp - with Snowpro Certification.Ent...Show moreLast updated: 30+ days ago
    • Promoted
    ▷ [Urgent] Lead Data Engineer

    ▷ [Urgent] Lead Data Engineer

    CoffeeBeansBengaluru, Karnataka, India
    Founded in the year 2017, CoffeeBeans specialises in offering high end consulting services in technology, product, and processes. We help our clients attain significant improvement in quality of del...Show moreLast updated: 10 days ago
    • Promoted
    Lead Data Engineer - Google Cloud Platform

    Lead Data Engineer - Google Cloud Platform

    Live Connections Placements Private LimitedBangalore
    About the Role : We are looking for a Lead GCP Data Engineer with strong expertise in Google Cloud Platform, PySpark, and SQL to design, build, and manage sca...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Searce IncBengaluru, IN
    Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 27 days ago
    • Promoted
    Kaleidofin - Lead Data Engineer - ETL / PySpark

    Kaleidofin - Lead Data Engineer - ETL / PySpark

    KaleidofinBangalore
    What youll do ? - You will take on the role of technical expert and a lead member of our Engineering team.Designing and building data pipelines from data ingestion to...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Veraxionvijayapura, India
    Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions. Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 6 days ago
    • Promoted
    Lead GCP Data Engineer

    Lead GCP Data Engineer

    ImpetusBengaluru, Karnataka, India
    Lead Data Engineer – GCP (BigQuery • Composer • Python • PySpark).You will lead the design, build and operation of large-scale data platforms on the Google Cloud Platform.You will manage a team of ...Show moreLast updated: 5 days ago
    • Promoted
    Lead Data Engineer T500-20510

    Lead Data Engineer T500-20510

    ANSRBengaluru, Republic Of India, IN
    ANSR is hiring for one of its clients.Martens is an iconic British brand founded in 1960 in Northamptonshire.Produced originally for workers looking for tough, durable boots, the brand was quickly ...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    USEReadyhosur, tamil nadu, in
    Job Title : Senior Databricks Engineer.As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakeho...Show moreLast updated: 27 days ago
    • Promoted
    GCP Data Engineer

    GCP Data Engineer

    JRD Systemshosur, tamil nadu, in
    Collaborate with data architects, analysts, and business teams to define and maintain.Roadnet) and modernized data sources. Partner with governance and infrastructure teams to establish.Automate and...Show moreLast updated: 1 day ago
    • Promoted
    Manager, Data Engineer

    Manager, Data Engineer

    Saks Global IndiaBangalore Urban, Karnataka, India
    Data Engineer and Operations Lead.Saks Global is the largest multi-brand luxury retailer in the world, comprising Saks Fifth Avenue, Neiman Marcus, Bergdorf Goodman, Saks OFF 5TH, Last Call and Hor...Show moreLast updated: 15 days ago