Senior Data EngineerInsight Global • pune, maharashtra, in

No longer accepting applications

Senior Data Engineer

Insight Global • pune, maharashtra, in

7 days ago

Job description

Summary

The Senior Data Engineer is responsible for building and optimizing ETL / ELT pipelines that process terabytes of data daily across 186 data assets, implementing BigQuery datasets with enterprise-scale performance optimization, and creating the data quality monitoring and transparency dashboards that enable data owner self-service.

Required Qualifications

Google Cloud Platform Data Engineering

5+ years of data engineering experience with at least 2+ years focused on Google Cloud Platform
Strong proficiency with BigQuery including :
Advanced SQL for analytical queries (window functions, CTEs, complex joins)
Partitioning and clustering strategies for performance optimization
Materialized views, authorized views, and query optimization techniques
Cost optimization through efficient query design and storage management
Understanding of BigQuery architecture (slot allocation, shuffle operations, distributed execution)
Hands-on experience with Google Cloud Dataflow and Apache Beam :
Pipeline development in Python or Java
Batch and streaming data processing patterns
Performance tuning and resource optimization
Error handling and pipeline monitoring
Proficiency with Cloud Composer (Apache Airflow) :
DAG development and dependency management
Airflow operators (BigQueryOperator, DataflowOperator, custom operators)
Workflow orchestration for complex multi-step processes
Monitoring and troubleshooting failed workflows

ETL / ELT & Data Integration

Strong experience building production-grade ETL / ELT pipelines processing terabyte-scale data

Knowledge of data integration patterns (full refresh, incremental load, change data capture)

Experience with data transformation techniques (normalization, denormalization, aggregation)

Understanding of data quality frameworks and validation strategies

Proficiency with schema evolution handling changes without breaking downstream systems

Experience with data lineage tracking from source to consumption

SQL & Database Technologies

Expert-level SQL skills including advanced analytics functions and query optimization

Understanding of database performance tuning (indexing, partitioning, query plans)

Experience with relational databases (PostgreSQL, MySQL, SQL Server) for source system integration

Familiarity with NoSQL databases (Firestore, Bigtable) for specialized use cases

Knowledge of data warehousing concepts (fact tables, dimension tables, slowly changing dimensions)

Programming & Scripting

Strong Python proficiency for data pipeline development and scripting

Experience with Apache Beam SDK for Dataflow pipeline development

Proficiency with Pandas, NumPy for data manipulation and analysis

Understanding of object-oriented programming and software engineering best practices

Experience with Git for version control and collaborative development

Basic Shell scripting for automation and operational tasks

Data Security & Compliance

Understanding of row-level security implementation patterns in BigQuery

Experience with PHI / PII data handling and healthcare compliance requirements (HIPAA preferred)

Knowledge of data masking and de-identification techniques

Understanding of audit logging and compliance reporting requirements

Familiarity with least privilege principles and data access controls

Preferred Qualifications

Google Cloud Professional Data Engineer certification

Healthcare industry experience with understanding of clinical and administrative data

Experience with Google Cloud Storage lifecycle policies and storage class optimization

Knowledge of Cloud Spanner for transactional workloads

Familiarity with Cloud DLP API for automated data classification

Experience with dbt (data build tool) for analytics engineering

Understanding of data mesh or data fabric architectural patterns

Background in DevOps practices and CI / CD for data pipelines

Experience with Terraform for infrastructure as code

Knowledge of data visualization tools (Looker, Tableau, Power BI)

Familiarity with machine learning workflows on GCP (Vertex AI)

Experience with Docker and Kubernetes (GKE) for containerized workloads

Create a job alert for this search

Senior Data Engineer • pune, maharashtra, in

Related jobs

Senior Data Engineer

EXL • Pune, Maharashtra, India

Key Responsibilities : • Develop high quality, secure and scalable data pipelines using spark, Scala / Python / Java on Hadoop or object storage like MinIO. Leverage technologies and solutions to innovat...Show more

Last updated: 30+ days ago • Promoted

Senior Data Engineer

Innova Solutions • Pune, Maharashtra, India

We're Hiring : Lead Data Consultant / Senior Data Engineer.Industry : Financial Services & Insurance (FSI).Are you passionate about building scalable, cloud-native data platforms? Join our team to le...Show more

Last updated: 16 days ago • Promoted

Senior Data Engineer

American Inference • Pune, IN

Please apply directly through for this position.We kindly request that candidates refrain from contacting company officials via email or messages regarding this role. We are an AI and Data Consulti...Show more

Last updated: 14 days ago • Promoted

Senior DataOps Engineer (AWS)

MSBC Group • Pune, IN

Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services.Lead the E-Comms data pipeline within Compass’s Application Simplification workstream : design, ...Show more

Last updated: 3 hours ago • Promoted • New!

Senior Data Engineer

Adaptive Technology Insights • Pune, IN

We are looking for an experienced.You will be responsible for designing, building, and optimizing data pipelines and architecture to support analytics, reporting, and data-driven decision-making ac...Show more

Last updated: 3 hours ago • Promoted • New!

Senior Data Engineer

Persistent Systems • Pune, Maharashtra, India

A highly skilled Data Engineering professional with 6 to 8 years of experience in designing, developing, and maintaining robust data solutions. Proven ability to work across the full software develo...Show more

Last updated: 30+ days ago • Promoted

Senior AWS Data Engineer

CYAN360 • Pune, IN

Position : Senior AWS Data Engineer.Work Timings : 2 : 30 PM to 11 : 30 PM IST.Need someone who can join immediately or in 15 days • • •. Design, develop, and deploy end-to-end data pipelines on AWS cloud in...Show more

Last updated: 30+ days ago • Promoted

Senior Data Engineer (Snowflake+ DBT)

Anblicks • pune, maharashtra, in

Senior Data Engineer (Snowflake + dbt).Hyderabad / Ahmedabad (Hybrid).Great Place to Work® Certified | Snowflake Elite Partner. We are looking for a highly skilled.ELT pipelines and analytical data ...Show more

Last updated: 2 days ago • Promoted

(Senior) Azure Data Engineer - Full remote - contractor in USD

All European Careers • pune, maharashtra, in

Remote

For an international project in Chennai, we are urgently looking for a Full Remote Senior Azure Data Engineer, who will build data pipeline for enterprise search applications using ADF and Databric...Show more

Last updated: 2 days ago • Promoted

Senior Data Engineer / Architect - Snowflake

Aezion, Inc • Pune, IN

We are seeking a highly skilled Snowflake Data Engineer with expertise in DBT, Airbyte, and Airflow, combined with strong experience in Kimball dimensional modeling. The ideal candidate will design ...Show more

Last updated: 3 hours ago • Promoted • New!

Senior Data Engineer

Donyati • Pune, IN

We are seeking a highly skilled Senior Data Engineer to join our team in building a modern data platform on AWS.You will play a key role in transitioning from legacy systems to a scalable, cloud-na...Show more

Last updated: 2 days ago • Promoted

Senior Data Engineer

SAIVA AI • Pune, IN

We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show more

Last updated: 30+ days ago • Promoted

Senior Data Engineer ( Palantir)

Lear Corporation • Pune, Maharashtra, India

Lear, a global automotive technology leader in Seating and E-Systems, is Making every drive better by delivering intelligent in-vehicle experiences for customers around the world.With over 100 year...Show more

Last updated: 13 days ago • Promoted

Senior Data Engineer

eClerx • Pune, Maharashtra, India

We are looking for an experienced Senior Data Engineer of hands-on experience in designing, building, and managing complex data architectures and BI solutions. The ideal candidate will have deep exp...Show more

Last updated: 6 days ago • Promoted

Senior Data Engineer

TEKsystems Global Services in India • Pune, IN

Develop and manage data pipelines using ADF and Azure data bricks.Expert knowledge in data warehousing, data modeling and ETL. Good to have datastage exposure.Expertise in on-prem to azure cloud mig...Show more

Last updated: 3 hours ago • Promoted • New!

Senior Data Engineer

Mindlance • Pune, IN

We’re looking for a strong, hands-on Sr Data Engineer who can independently drive client conversations, collaborate with SMEs, and deliver high-quality data solutions in a cloud-native environment....Show more

Last updated: 3 hours ago • Promoted • New!

5447 | Senior Data Engineer

EXL • Pune, Maharashtra, India

We are seeking a Senior Data Engineer to build and optimize data systems that power batch processing, real-time streaming, pipeline orchestration, data lake management, and data cataloging.You will...Show more

Last updated: 24 days ago • Promoted

Senior Data Engineer

SPIRO • Pune, Maharashtra, India

Position : Senior Data Engineer Location : Pune Experience : 5+ years Job Summary : We are seeking an experienced Senior Data Engineer with a strong background in PySpark, Spark, and Big Data techn...Show more

Last updated: 30+ days ago • Promoted