Consultant – Cloud Data Engineer – GCP

Key Responsibilities :

• Design, develop, and manage scalable, secure, and efficient data pipelines on Google Cloud Platform (GCP) to process and transform large datasets from multiple sources.
• Implement and optimize GCP data services such as BigQuery, Cloud Storage, Cloud SQL, Dataflow, Dataproc, Pub / Sub, and Cloud Functions.
• Architect and maintain ETL / ELT processes to ensure efficient data ingestion, transformation, and storage using Cloud Data Fusion, Dataflow (Apache Beam), or Composer (Airflow).
• Architect and implement data models on GCP to support efficient data storage and retrieval in BigQuery, Cloud Spanner, or Firestore.
• Collaborate with data architects to design and implement data lakes, data warehouses, and data marts on GCP.
• Build and maintain data integration workflows using tools like Cloud Composer (Apache Airflow), Cloud Functions, or Dataflow.
• Utilize GCP DevOps tools for source control, build automation, release management, and infrastructure as code (IaC) using Terraform or Deployment Manager.
• Implement CI / CD pipelines within Google Cloud Build to automate the build, test, and deployment of data pipelines and infrastructure changes, ensuring rapid and reliable delivery of data solutions.
• Utilize big data technologies such as Dataproc (Apache Spark, Hadoop, etc.) to handle large volumes of data and perform complex analytics tasks.
• Implement real-time data processing solutions using streaming technologies like Pub / Sub and Dataflow (Apache Beam) to enable timely insights and actions.
• Implement data governance policies and security controls within GCP environments to protect sensitive data and ensure compliance with regulatory requirements such as GDPR, HIPAA, or PCI DSS.
• Optimize data pipelines and processing workflows for performance, scalability, and cost-effectiveness on GCP, leveraging BigQuery optimizations, Autoscaling, and cost management techniques.
• Monitor, troubleshoot, and optimize the performance of data pipelines and cloud-based systems to ensure high availability, low latency, and scalability.
• Work closely with cross-functional teams to understand business requirements and translate them into scalable GCP data solutions.
• Ensure data security and compliance with industry best practices, including IAM roles, VPC configurations, encryption, and access controls.
• Conduct regular code reviews, provide feedback, and ensure adherence to best practices and standards.
• Stay up to date with the latest GCP services and cloud technologies to drive innovation within the team.

Job Qualifications :