Talent.com
This job offer is not available in your country.
[Immediate Start] Lead Data Engineer- GCP

[Immediate Start] Lead Data Engineer- GCP

EgenHyderabad, Telangana, India
4 hours ago
Job description

Job Overview :

We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team. The ideal candidate will be responsible for requirements gathering, designing, architecting the solution, developing, and maintaining robust and scalable ETL (Extract, Transform, Load) & ELT data pipelines. The role involves working with customers directly, gathering requirements, discovery phase, designing, architecting the solution, using various GCP services, implementing data transformations, data ingestion, data quality, and consistency across systems, and post post-delivery support.

Experience Level : 10 to 12 years of relevant IT experience

Key Responsibilities :

  • Design, develop, test, and maintain scalable ETL data pipelines using Python.
  • Architect the enterprise solutions with various technologies like Kafka, multi-cloud services, auto-scaling using GKE, Load balancers, APIGEE proxy API management, DBT, using LLMs as needed in the solution, redaction of sensitive information, DLP (Data Loss Prevention) etc.
  • Work extensively on Google Cloud Platform (GCP) services such as :
  • Dataflow for real-time and batch data processing
  • Cloud Functions for lightweight serverless compute
  • BigQuery for data warehousing and analytics
  • Cloud Composer for orchestration of data workflows (on Apache Airflow)
  • Google Cloud Storage (GCS) for managing data at scale
  • IAM for access control and security
  • Cloud Run for containerized applications
  • Should have experience in the following areas :
  • API framework : Python FastAPI
  • Processing engine : Apache Spark
  • Messaging and streaming data processing : Kafka
  • Storage : MongoDB, Redis / Bigtable
  • Orchestration : Airflow
  • Experience in deployments in GKE, Cloud Run.
  • Perform data ingestion from various sources and apply transformation and cleansing logic to ensure high-quality data delivery.
  • Implement and enforce data quality checks, validation rules, and monitoring.
  • Collaborate with data scientists, analysts, and other engineering teams to understand data needs and deliver efficient data solutions.
  • Manage version control using GitHub and participate in CI / CD pipeline deployments for data projects.
  • Write complex SQL queries for data extraction and validation from relational databases such as SQL Server, Oracle, or PostgreSQL.
  • Document pipeline designs, data flow diagrams, and operational support procedures.

Required Skills :

  • 10 to 12 years of hands-on experience in Python for backend or data engineering projects.
  • Strong understanding and working experience with GCP cloud services (especially Dataflow, BigQuery, Cloud Functions, Cloud Composer, etc.).
  • Solid understanding of data pipeline architecture, data integration, and transformation techniques.
  • Experience in working with version control systems like GitHub and knowledge of CI / CD practices.
  • Experience in Apache Spark, Kafka, Redis, Fast APIs, Airflow, GCP Composer DAGs.
  • Strong experience in SQL with at least one enterprise database (SQL Server, Oracle, PostgreSQL, etc.).
  • Experience in data migrations from on-premise data sources to Cloud platforms.
  • Good to Have (Optional Skills) :

  • Experience working with the Snowflake cloud data platform.
  • Hands-on knowledge of Databricks for big data processing and analytics.
  • Familiarity with Azure Data Factory (ADF) and other Azure data engineering tools.
  • Additional Details :

  • Excellent problem-solving and analytical skills.
  • Strong communication skills and ability to collaborate in a team environment.
  • Education :

  • Bachelor's degree in Computer Science, a related field, or equivalent experience.
  • Create a job alert for this search

    Gcp Data Engineer • Hyderabad, Telangana, India

    Related jobs
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Searce IncHyderabad, IN
    Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 6 days ago
    • Promoted
    Data Engineer Lead

    Data Engineer Lead

    Enterprise MindsHyderabad, Telangana, India
    We’re Hiring : Data Engineer Lead at Enterprise Minds #Hiring #DataEngineer #Cloud #Azure #AWS #GCP #HyderabadJobs #EnterpriseMinds #Careers] Role : Data Engineer Lead Experience : 8–12 years Loca...Show moreLast updated: 16 days ago
    • Promoted
    Data Engineer - Google Cloud Platform

    Data Engineer - Google Cloud Platform

    TwinPacs Sdn BhdHyderabad
    Were Hiring : GCP Data Engineer (Senior / Lead Programmer Analyst) Location : Hyderabad (Work from Office - Q City) Mo...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    ITQube LTDsecunderabad, telangana, in
    Looking for a Data Engineer to join our engineering will contribute directly to the design, automation, and optimization of our data processes, primarily developing solutions in Python within the A...Show moreLast updated: 6 days ago
    • Promoted
    Lead Snowflake Data Engineer

    Lead Snowflake Data Engineer

    Quess Corp Limitedhyderabad, telangana, in
    We are seeking an experienced Lead Snowflake Data Engineer to join our Data & Analytics team.This role involves designing, implementing, and optimizing Snowflake-based data solutions while providin...Show moreLast updated: 4 days ago
    • Promoted
    Lead GCP Data Engineer

    Lead GCP Data Engineer

    Incedo Inc.Hyderabad, Telangana, India
    Lead the end-to-end design and implementation of.Architect scalable data platforms leveraging.BigQuery, Dataflow, Pub / Sub, Dataproc, Cloud Storage, Composer, and Cloud Functions.CI / CD, version cont...Show moreLast updated: 18 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    EverestDX IncHyderabad, Telangana, India
    Everest DX – We are a Digital Platform Services company, headquartered in Stamford.Our Platform / Solution includes Orchestration, Intelligent operations with BOTs’, AI-powered analytics for Enterpri...Show moreLast updated: 6 days ago
    • Promoted
    GCP Big Data Engineer

    GCP Big Data Engineer

    Talentmaticshyderabad, telangana, in
    The ideal candidate will bring strong leadership capabilities, technical depth, and a proven track record of delivering end-to-end big data solutions in cloud environments.Lead and mentor teams in ...Show moreLast updated: 16 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Prudent Technologies and Consulting, Inc.Hyderabad, Telangana, India
    Seeking an experienced Senior Data Engineer with over 9 years of industry experience to architect, build, and manage advanced data pipelines and platforms in a cloud environment.The ideal candidate...Show moreLast updated: 6 days ago
    • Promoted
    • New!
    Azure Data Engineer [Immediate Start]

    Azure Data Engineer [Immediate Start]

    Tata Consultancy ServicesHyderabad, Telangana, India
    Greetings from TCS! Job Title : Azure Data Engineer Required Skillset : Azure Databricks(Strong),scala(strong), Pyspark(Good to have) Location : PAN INDIA Experience Range : 4-10 years Job Descrip...Show moreLast updated: less than 1 hour ago
    • Promoted
    GCP Data Engineer

    GCP Data Engineer

    TEKsystemsHyderabad, IN
    We are actively hiring for Skilled GCP Data Engineers for a global banking client.Years of experience : 4+ years (relevant). Looking for immediate joiners • •.Onboard new data sources - negotiate, agre...Show moreLast updated: 19 days ago
    • Promoted
    Lead Data Engineer- GCP

    Lead Data Engineer- GCP

    EgenHyderabad, Telangana, India
    We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team.The ideal candidate will...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    AnblicksHyderabad, Telangana, India
    Location : Hyderabad or Ahmedabad.Skills : Snowflake, Python / Pyspark, SQL.Only Immediate to 15 Days joiners apply.Lead the end-to-end Snowflake platform implementation, including architecture, design...Show moreLast updated: 30+ days ago
    • Promoted
    Bigdata Engineer

    Bigdata Engineer

    TAO Digital SolutionsHyderabad, India
    Highly skilled Big Data Engineer with expertise in distributed systems, and advanced programming techniques.Candidate will possess experience in developing highly scalable Big Data pipelines, stron...Show moreLast updated: 17 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Eucloid Data SolutionsHyderabad, IN
    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 19 days ago
    • Promoted
    Data Engineer - Databricks

    Data Engineer - Databricks

    Tata Consultancy ServicesHyderabad, Telangana, India
    TCS is hiring Data Engineer (Databricks).Years of Experience – 10-16 years.Senior Data Engineer – Databricks.Databricks, PySpark, Delta Lake, MLflow, Azure / AWS / GCP, Data Lakehouse architecture, CI / ...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer (GCP)

    Data Engineer (GCP)

    TalescopeHyderabad, India
    We are looking for a highly experienced and hands-on GCP Data Engineer (AdTech) to lead our analytics team.This role is ideal for someone with a strong background in log-level data handling, cros...Show moreLast updated: 6 days ago
    • Promoted
    • New!
    [Immediate Start] Bigdata Engineer

    [Immediate Start] Bigdata Engineer

    TAO Digital SolutionsHyderabad, Telangana, India
    Immediate Joiners! Job Description : Highly skilled Big Data Engineer with expertise in distributed systems, and advanced programming techniques. Candidate will possess experience in developing hig...Show moreLast updated: less than 1 hour ago
    • Promoted
    Lead Data Engineer [T500-20180]

    Lead Data Engineer [T500-20180]

    InspireHyderabad, Telangana, India
    Inspire Brands is disrupting the restaurant industry through digital transformation and operational efficiencies.The company’s technology hub, Inspire Brands Hyderabad Support Center, India, will l...Show moreLast updated: 28 days ago
    • Promoted
    GenAI Lead Engineer (Investment Data Platforms)

    GenAI Lead Engineer (Investment Data Platforms)

    Vichara Technologieshyderabad, telangana, in
    We are seeking a highly skilled.This role will also involve leading a cross-functional team of.Machine Learning Engineers and UI Developers. Develop custom frameworks using.Optimize LLM usage for in...Show moreLast updated: 30+ days ago