Senior Data Engineer

Insight GlobalKanpur, IN

30+ days ago

Job description

Summary

The Senior Data Engineer is responsible for building and optimizing ETL / ELT pipelines that process terabytes of data daily across 186 data assets, implementing BigQuery datasets with enterprise-scale performance optimization, and creating the data quality monitoring and transparency dashboards that enable data owner self-service.

Required Qualifications

Google Cloud Platform Data Engineering

5+ years of data engineering experience with at least 2+ years focused on Google Cloud Platform
Strong proficiency with BigQuery including :
Advanced SQL for analytical queries (window functions, CTEs, complex joins)
Partitioning and clustering strategies for performance optimization
Materialized views, authorized views, and query optimization techniques
Cost optimization through efficient query design and storage management
Understanding of BigQuery architecture (slot allocation, shuffle operations, distributed execution)
Hands-on experience with Google Cloud Dataflow and Apache Beam :
Pipeline development in Python or Java
Batch and streaming data processing patterns
Performance tuning and resource optimization
Error handling and pipeline monitoring
Proficiency with Cloud Composer (Apache Airflow) :
DAG development and dependency management
Airflow operators (BigQueryOperator, DataflowOperator, custom operators)
Workflow orchestration for complex multi-step processes
Monitoring and troubleshooting failed workflows

ETL / ELT & Data Integration

Strong experience building production-grade ETL / ELT pipelines processing terabyte-scale data

Knowledge of data integration patterns (full refresh, incremental load, change data capture)

Experience with data transformation techniques (normalization, denormalization, aggregation)

Understanding of data quality frameworks and validation strategies

Proficiency with schema evolution handling changes without breaking downstream systems

Experience with data lineage tracking from source to consumption

SQL & Database Technologies

Expert-level SQL skills including advanced analytics functions and query optimization

Understanding of database performance tuning (indexing, partitioning, query plans)

Experience with relational databases (PostgreSQL, MySQL, SQL Server) for source system integration

Familiarity with NoSQL databases (Firestore, Bigtable) for specialized use cases

Knowledge of data warehousing concepts (fact tables, dimension tables, slowly changing dimensions)

Programming & Scripting

Strong Python proficiency for data pipeline development and scripting

Experience with Apache Beam SDK for Dataflow pipeline development

Proficiency with Pandas, NumPy for data manipulation and analysis

Understanding of object-oriented programming and software engineering best practices

Experience with Git for version control and collaborative development

Basic Shell scripting for automation and operational tasks

Data Security & Compliance

Understanding of row-level security implementation patterns in BigQuery

Experience with PHI / PII data handling and healthcare compliance requirements (HIPAA preferred)

Knowledge of data masking and de-identification techniques

Understanding of audit logging and compliance reporting requirements

Familiarity with least privilege principles and data access controls

Preferred Qualifications

Google Cloud Professional Data Engineer certification

Healthcare industry experience with understanding of clinical and administrative data

Experience with Google Cloud Storage lifecycle policies and storage class optimization

Knowledge of Cloud Spanner for transactional workloads

Familiarity with Cloud DLP API for automated data classification

Experience with dbt (data build tool) for analytics engineering

Understanding of data mesh or data fabric architectural patterns

Background in DevOps practices and CI / CD for data pipelines

Experience with Terraform for infrastructure as code

Knowledge of data visualization tools (Looker, Tableau, Power BI)

Familiarity with machine learning workflows on GCP (Vertex AI)

Experience with Docker and Kubernetes (GKE) for containerized workloads

Create a job alert for this search

Senior Data Engineer • Kanpur, IN

Related jobs

Promoted

Senior Data Engineer

SAIVA AIKanpur, IN

We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer - Data Acquisition

InfoBeansKanpur, IN

We are seeking a highly skilled.Senior Data Engineer – Data Acquisition (ODS).The ideal candidate will have extensive hands-on experience in building and optimizing data ingestion and transformatio...Show moreLast updated: 1 day ago

Promoted

Senior Data Engineer

Saras Analyticskanpur, uttar pradesh, in

We are an ecommerce focused end to end data analytics firm assisting enterprises & brands in data driven decision making to maximize business value. Our suite of work spans extraction, transformatio...Show moreLast updated: 30+ days ago

Promoted

Senior Data Integration Engineer

IndsafriKanpur, IN

Senior Data Integration Engineer.We are seeking a highly experienced and strategic Senior Data Integration Engineer to architect, build, and manage the data pipelines that power our customer intell...Show moreLast updated: 29 days ago

Promoted

Senior Data Engineer

Incept Labskanpur, uttar pradesh, in

Incept Labs is an AI research lab based in San Francisco, California.Our small team of scientists, engineers, and builders is passionate about developing domain-specific, next-generation AI solutio...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

HealthyMe AIKanpur, IN

We are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data.You will report to the Head of AI and Engineering and be responsible for designi...Show moreLast updated: 30+ days ago

Promoted

Senior Data Platform Engineer

Black Dog LabsKanpur, IN

Remote (collaboration across time zones), India or LATAM preferred.Proficient English communication.Data Engineering / Backend Engineering / DevOps. We’re looking for a hands-on Senior Data Platform...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer-Remote

Cimpress Indiakanpur, uttar pradesh, in

Remote

NASDAQ : CMPR) is the world leader in mass customisation.Our unmatched technology, production, and supply chain operations allow us to offer products that can be personalised by an individual custom...Show moreLast updated: 28 days ago

Promoted

Data Engineer

thinkbridgekanpur, uttar pradesh, in

We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 30+ days ago

Promoted

Senior AWS Data Engineer

CYAN360Kanpur, IN

Position : Senior AWS Data Engineer.Work Timings : 2 : 30 PM to 11 : 30 PM IST.Need someone who can join immediately or in 15 days • • •. Design, develop, and deploy end-to-end data pipelines on AWS cloud in...Show moreLast updated: 30+ days ago

Promoted

Azure Senior Data Engineer

Tata Consultancy Serviceskanpur, uttar pradesh, in

TCS is hiring for AWS Data Engineer.Extensive coding experience including Python / Pyspark.Experience developing in Azure with key data technologies (e. ADLS, ADF, Azure Databricks etc.Software develo...Show moreLast updated: 3 days ago

Promoted

Senior Data Engineer

IntelliasKanpur, IN

Apache Flink / Apache Spark (Streaming).Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines. Kafka + Flink / Spark Streaming).Python for data engin...Show moreLast updated: 8 days ago

Promoted

Data Engineer

Insight GlobalKanpur, IN

GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts. Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

Straivekanpur, uttar pradesh, in

The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

Mastekkanpur, uttar pradesh, in

We are looking for a Senior Data Engineer with good exposure to Cloud Services in AWS(IAM, RDS, Lambda, DMS) along with Snowflake / Postgresql, ETL and Streaming Services like Kafka.Show moreLast updated: 30+ days ago

Promoted

Snowflake Data Engineer

Newpage SolutionsKanpur, IN

Location : Remote | Type : Contract.Newpage Solutions is a global digital health innovation company helping people live longer, healthier lives. We partner with life sciences organizations—including p...Show moreLast updated: 8 days ago

Promoted

Senior AWS Data Engineer_Exp : 5+ Years

Atyeti Inckanpur, uttar pradesh, in

Required Skills & Qualifications.Bachelor’s or Master’s degree in Computer Science or equivalent experience.Application Developer or in similar software engineering roles.Python; strong SQL and clo...Show moreLast updated: 8 days ago

Promoted

Data Engineer

DigitalzoneKanpur, IN

As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics. Develop and maintain scalable ETL / ELT pipelines using Py...Show moreLast updated: 8 days ago