Talent.com
This job offer is not available in your country.
D-TechWorks - Lead Data Engineer - Google Cloud Platform

D-TechWorks - Lead Data Engineer - Google Cloud Platform

D-TechWorks Pvt LtdPune
16 days ago
Job description

Responsibilities :

  • Lead and mentor a team of data engineers, providing technical guidance, setting best practices, and overseeing task execution for the migration project.
  • Design, develop, and architect scalable ETL processes to extract, transform, and load petabytes of data from on-premises SQL Server to GCP Cloud SQL PostgreSQL.
  • Oversee the comprehensive analysis of existing SQL Server schemas, data types, stored procedures, and complex data models, defining strategies for their optimal conversion and refactoring for PostgreSQL.
  • Establish and enforce rigorous data validation, quality, and integrity frameworks throughout the migration lifecycle, ensuring accuracy and consistency.
  • Collaborate strategically with Database Administrators, application architects, business stakeholders, and security teams to define migration scope, requirements, and cutover plans.
  • Lead the development and maintenance of advanced scripts (primarily Python) for automating large-scale migration tasks, complex data transformations, and reconciliation processes.
  • Proactively identify, troubleshoot, and lead the resolution of complex data discrepancies, performance bottlenecks, and technical challenges during migration.
  • Define and maintain comprehensive documentation standards for migration strategies, data mapping, transformation rules, and post-migration validation procedures.
  • Ensure data governance, security, and compliance standards are meticulously applied throughout the migration process, including data encryption and access controls within GCP.
  • Implement Schema conversion or custom schema mapping strategy for SQL Server to PostgreSQL shift
  • Refactor and translate complex stored procedures and T-SQL logic to PostgreSQL-compatible constructs while preserving functional equivalence.
  • Develop and execute comprehensive data reconciliation strategies to ensure consistency and parity between legacy and migrated datasets post-cutover.
  • Design fallback procedures and lead post-migration verification and support to ensure business continuity.
  • Ensuring metadata cataloging and data lineage tracking using GCP-native or integrated tools.

Must-Have Skills :

  • Expertise in data engineering, specifically for Google Cloud Platform (GCP).
  • Deep understanding of relational database architecture, advanced schema design, data modeling, and performance tuning.
  • Expert-level SQL proficiency, with extensive hands-on experience in both T-SQL (SQL Server) and PostgreSQL.
  • Hands-on experience with data migration processes, including moving datasets from on-premises databases to cloud storage solutions.
  • Proficiency in designing, implementing, and optimizing complex ETL / ELT pipelines for high-volume data movement, leveraging tools and custom scripting.
  • Strong knowledge of GCP services : Cloud SQL, Dataflow, Pub / Sub, Cloud Storage, Dataproc, Cloud Composer, Cloud Functions, and Bigquery.
  • Solid understanding of data governance, security, and compliance practices in the cloud, including the management of sensitive data during migration.
  • Strong programming skills in Python or Java for building data pipelines and automating processes.
  • Experience with real-time data processing using Pub / Sub, Dataflow, or similar GCP services.
  • Experience with CI / CD practices and tools like Jenkins, GitLab, or Cloud Build for automating the data engineering pipeline.
  • Knowledge of data modeling and best practices for structuring cloud data storage for optimal query performance and analytics in GCP.
  • Familiarity with observability and monitoring tools in GCP (e.g., Stackdriver, Prometheus) for real-time data pipeline visibility and alerting.

    • Good-to-Have Skills :

  • Direct experience with GCP Database Migration Service, Storage Transfer Service, or similar cloud-native migration tools.
  • Familiarity with data orchestration using tools like Cloud Composer (based on Apache Airflow) for managing workflows.
  • Experience with containerization tools like Docker and Kubernetes for deploying data pipelines in a scalable manner.
  • Exposure to DataOps tools and methodologies for managing data workflows.
  • Experience with machine learning platforms like AI Platform in GCP to integrate with data pipelines.
  • Familiarity with data lake architecture and the integration of BigQuery with Google Cloud Storage or Dataproc.

    • ref : hirist.tech)

    Create a job alert for this search

    Cloud Engineer • Pune