Gcp Engineer

Job description

JD :

1.  Data Pipeline Development & Orchestration  -Design, develop, and orchestrate data pipelines using Apache Airflow (Cloud Composer) to automate data ingestion, transformation, and loading (ETL / ELT) workflows.

2. Data Transformation with DBT - Implement DBT (Data Build Tool) models  for transforming raw data into analytics-ready datasets in BigQuery, applying SQL-based transformations, modular modeling, and version control.

3. Stream & Batch Processing with Dataflow - Build real-time (Streaming) and batch data processing pipelines using Apache Beam on Dataflow, ensuring scalable and efficient data processing.

4.  BigQuery Optimization & Performance Tuning  - Design optimized BigQuery schemas, implement partitioning, clustering, and tune queries for cost-efficiency and performance.

5.  Monitoring, Logging, and CI / CD for Data Pipelines  - Set up monitoring with Stackdriver, logging, error handling, and CI / CD pipelines for automated testing and deployment of DBT, Airflow DAGs, and Dataflow jobs.

Skills Required

Apache Airflow, BigQuery