Lead Data Engineer - Python & GCP

Job Title : Lead Data Engineer – Python & GCP Location : Hyderabad

Exp : 10+ Years

Overview :

We are seeking an experienced Lead Data Engineer with strong expertise in Python and Google Cloud Platform (GCP) . You will design, develop, and manage scalable ETL / ELT data pipelines , work closely with clients to understand requirements, and build data solutions using modern cloud and open-source technologies.

Key Responsibilities :

Design, build, and maintain scalable data pipelines using Python and GCP services .

Work with GCP tools such as Dataflow, BigQuery, Cloud Composer, Cloud Functions, Cloud Run, GCS, and IAM .

Implement data ingestion, transformation, and quality checks to ensure accurate, reliable data.

Architect enterprise data solutions using tools like Kafka, Spark, Airflow, DBT, GKE, APIGEE, and DLP .

Use FastAPI for API development and GitHub for version control.

Deploy solutions using CI / CD pipelines and manage data workflows in Airflow / Cloud Composer .

Write and optimize complex SQL queries (SQL Server, Oracle, PostgreSQL, etc.).

Collaborate with data scientists and analysts to deliver efficient data solutions.

Document data flows, pipelines, and operational procedures.

Required Skills & Experience :

10+ years of experience in data engineering or backend development.

Strong programming skills in Python .

Hands-on experience with GCP (Dataflow, BigQuery, Cloud Functions, Cloud Composer).

Experience with Kafka, Spark, Redis, Airflow, GKE, Cloud Run, and FastAPI .

Excellent knowledge of SQL and relational databases.

Experience with CI / CD and GitHub .

Experience migrating data from on-premise systems to the cloud.

Nice to Have :

Experience with Snowflake or Databricks .

Knowledge of Azure Data Factory or other Azure data tools.

Exposure to machine learning or LLMs in data solutions.