Gcp Engineer

Job description

JD:

1. Data Pipeline Development & Orchestration -Design, develop, and orchestrate data pipelines using Apache Airflow (Cloud Composer) to automate data ingestion, transformation, and loading (ETL/ELT) workflows.

2. Data Transformation with DBT -Implement DBT (Data Build Tool) models for transforming raw data into analytics-ready datasets in BigQuery, applying SQL-based transformations, modular modeling, and version control.

3. Stream & Batch Processing with Dataflow - Build real-time (Streaming) and batch data processing pipelines using Apache Beam on Dataflow, ensuring scalable and efficient data processing.

4. BigQuery Optimization & Performance Tuning - Design optimized BigQuery schemas, implement partitioning, clustering, and tune queries for cost-efficiency and performance.

5. Monitoring, Logging, and CI/CD for Data Pipelines - Set up monitoring with Stackdriver, logging, error handling, and CI/CD pipelines for automated testing and deployment of DBT, Airflow DAGs, and Dataflow jobs.

Skills Required
Apache Airflow, BigQuery