Palni - GCP Data Engineer - ETL

Key Responsibilities :

• Design and develop data pipelines and ETL processes to ingest, process, and store large volumes of data.
• Implement and manage big data technologies such as Kafka, Dataflow, BigQuery, CloudSQL, PubSub
• Collaborate with stakeholders to understand data requirements and deliver high-quality data solutions.
• Monitor and troubleshoot data pipeline issues and implement solutions to prevent future occurrences.

Required Skills and Experience :

Generally, we use Google Cloud Platform (GCP) for all software deployed at Wayfair.

Data Storage and Processing

BigQuery

CloudSQL

PostgreSQL

DataProc

Pub / Sub

Data modeling :

Breaking the business requirements(KPIs) to data points.

Building the scalable data model

ETL Tools :

DBT

SQL

Data Orchestration and ETL

Dataflow

Cloud Composer

Infrastructure and Deployment

Kubernetes

Helm

Data Access and Management

Looker

Terraform

Ideal Business Domain Experience :

Supply chain or warehousing experience

The project is focused on building a normalized data layer which ingests information from multiple Warehouse

Management Systems (WMS) and projects it for backoffice analysis.

(ref : hirist.tech)