No longer accepting applications

Senior Data Engineer

Insight Globalbangalore district, karnataka, in

10 days ago

Job description

Summary

The Senior Data Engineer is responsible for building and optimizing ETL / ELT pipelines that process terabytes of data daily across 186 data assets, implementing BigQuery datasets with enterprise-scale performance optimization, and creating the data quality monitoring and transparency dashboards that enable data owner self-service.

Required Qualifications

Google Cloud Platform Data Engineering

5+ years of data engineering experience with at least 2+ years focused on Google Cloud Platform
Strong proficiency with BigQuery including :
Advanced SQL for analytical queries (window functions, CTEs, complex joins)
Partitioning and clustering strategies for performance optimization
Materialized views, authorized views, and query optimization techniques
Cost optimization through efficient query design and storage management
Understanding of BigQuery architecture (slot allocation, shuffle operations, distributed execution)
Hands-on experience with Google Cloud Dataflow and Apache Beam :
Pipeline development in Python or Java
Batch and streaming data processing patterns
Performance tuning and resource optimization
Error handling and pipeline monitoring
Proficiency with Cloud Composer (Apache Airflow) :
DAG development and dependency management
Airflow operators (BigQueryOperator, DataflowOperator, custom operators)
Workflow orchestration for complex multi-step processes
Monitoring and troubleshooting failed workflows

ETL / ELT & Data Integration

Strong experience building production-grade ETL / ELT pipelines processing terabyte-scale data

Knowledge of data integration patterns (full refresh, incremental load, change data capture)

Experience with data transformation techniques (normalization, denormalization, aggregation)

Understanding of data quality frameworks and validation strategies

Proficiency with schema evolution handling changes without breaking downstream systems

Experience with data lineage tracking from source to consumption

SQL & Database Technologies

Expert-level SQL skills including advanced analytics functions and query optimization

Understanding of database performance tuning (indexing, partitioning, query plans)

Experience with relational databases (PostgreSQL, MySQL, SQL Server) for source system integration

Familiarity with NoSQL databases (Firestore, Bigtable) for specialized use cases

Knowledge of data warehousing concepts (fact tables, dimension tables, slowly changing dimensions)

Programming & Scripting

Strong Python proficiency for data pipeline development and scripting

Experience with Apache Beam SDK for Dataflow pipeline development

Proficiency with Pandas, NumPy for data manipulation and analysis

Understanding of object-oriented programming and software engineering best practices

Experience with Git for version control and collaborative development

Basic Shell scripting for automation and operational tasks

Data Security & Compliance

Understanding of row-level security implementation patterns in BigQuery

Experience with PHI / PII data handling and healthcare compliance requirements (HIPAA preferred)

Knowledge of data masking and de-identification techniques

Understanding of audit logging and compliance reporting requirements

Familiarity with least privilege principles and data access controls

Preferred Qualifications

Google Cloud Professional Data Engineer certification

Healthcare industry experience with understanding of clinical and administrative data

Experience with Google Cloud Storage lifecycle policies and storage class optimization

Knowledge of Cloud Spanner for transactional workloads

Familiarity with Cloud DLP API for automated data classification

Experience with dbt (data build tool) for analytics engineering

Understanding of data mesh or data fabric architectural patterns

Background in DevOps practices and CI / CD for data pipelines

Experience with Terraform for infrastructure as code

Knowledge of data visualization tools (Looker, Tableau, Power BI)

Familiarity with machine learning workflows on GCP (Vertex AI)

Experience with Docker and Kubernetes (GKE) for containerized workloads

Create a job alert for this search

Senior Data Engineer • bangalore district, karnataka, in

Related jobs

Promoted

Senior Data Engineer

TEKsystems Global Services in IndiaBengaluru, Karnataka, India

Develop and manage data pipelines using ADF and Azure data bricks.Expert knowledge in data warehousing, data modeling and ETL. Good to have datastage exposure.Expertise in on-prem to azure cloud mig...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

ZluriBengaluru, Karnataka, India

Job Location : Whitefield, Bangalore.Reporting to : Engineering Manager.Identity Governance and Administration (IGA) platform that enables IT and security teams to discover identities and applicati...Show moreLast updated: 15 days ago

Promoted

Senior Data Engineer

Nityo InfotechBengaluru, Karnataka, India

We are seeking an experienced Senior Data Engineer with 8-12 years of hands-on experience to join our team on-site.This role requires a self-starter who can architect and implement robust data foun...Show moreLast updated: 18 days ago

Promoted

Senior Data Engineer

MindlanceBangalore, IN

We’re looking for a strong, hands-on Sr Data Engineer who can independently drive client conversations, collaborate with SMEs, and deliver high-quality data solutions in a cloud-native environment....Show moreLast updated: 2 days ago

Promoted

Senior Data Engineer

CargillBengaluru, Karnataka, India

The Senior Professional, Data Engineering designs, builds, and maintains complex data systems that enable data analysis and reporting. This role includes a focus on Treasury-specific data solutions,...Show moreLast updated: 9 days ago

Promoted

Senior Data Engineer

VeraxionBengaluru, Karnataka, India

We are looking for a Senior Data Engineer who can design, build, and scale modern data platforms that power analytics, decision-making, and AI across the organization. If you love solving complex da...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

Delta Air LinesBengaluru, India

Design, architect, and optimize Teradata EDW solutions with a focus on performance, scalability, and maintainability.Lead performance tuning efforts using TASM and other workload management tools.D...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

Relevance LabBengaluru, Karnataka, India

Design, build and maintain enterprise data models and data assets on Microsoft Azure to support analytics, reporting, ML / AI and data products. Translate business requirements into robust logical and...Show moreLast updated: 3 days ago

Promoted
New!

Data Engineer I

LiciousGreater Bengaluru Area, India

SDE 1 – Data Platform Engineer.We are looking for a passionate and motivated.Software Development Engineer (SDE 1) – Data Platform. Data Platform Engineering team at.In this role, you’ll design, bui...Show moreLast updated: 15 hours ago

Promoted

Senior Data Engineer

Live Connectionshosur, tamil nadu, in

Hiring : Data Engineer | 9-12 Years Experience | Remote (India).Python, SQL, Big Data, Cloud, Snowflake.We are looking for an experienced. The ideal candidate will have deep experience in.Python, SQL...Show moreLast updated: 30+ days ago

Promoted
New!

Senior AI Engineer

Zingly.aiGreater Bengaluru Area, India

Help us reinvent conversations between people and brands.At Zingly, we’re not just building another chatbot—we’re building relationship tech. As a fast-moving, post-seed, pre-Series A startup, our m...Show moreLast updated: 15 hours ago

Promoted

Senior Data Engineer

ACL DigitalBengaluru, Karnataka, India

We are looking for professionals with expertise in Data Engineering.Data Engineer (We are looking for 7 to 10years of experience) and Tech Lead – Data Engineer -( looking for 10+ years of experienc...Show moreLast updated: 16 days ago

Promoted

Senior Data Engineer

ADMBengaluru, India

Do you want to join a team passionate about engineering data solutions to inform business decisions and transform business performance? ADM is seeking a Sr. Data Engineer, someone who thrives in an ...Show moreLast updated: 17 days ago

Promoted

Senior Data Engineer

CloudwickBengaluru, Karnataka, India

The ideal fit for the role will have a strong background in handling large volumes of data with Apache Spark to build , enhance bespoke systems to harness data. The key focus of the position is to b...Show moreLast updated: 9 days ago

Promoted

Senior Azure Data Engineer

Tata Consultancy ServicesGreater Bengaluru Area, India

Job Title : Senior Azure Data Engineer.Required Skillset : Azure data factory, data bricks, data lake, automation and performance optimization of ETL. Good Knowledge of Data Brick lakehouse and Azure ...Show moreLast updated: 30+ days ago

Promoted
New!

Senior Data Engineer

ArcanaGreater Bengaluru Area, India

Join our dynamic team at the forefront of cutting-edge technology as we seek a seasoned Senior Data Engineer.Embark on a journey where your deep-rooted expertise in computer science fundamentals, a...Show moreLast updated: 15 hours ago

Promoted

Senior Data Engineer

Pretium Enterprise Services, IndiaBengaluru, Karnataka, India

Would be a lead responsible for developing data products & solutions for the organization.IT with experience in Data warehouse, ETL, BI projects is required. Would collaborate within Agile teams to ...Show moreLast updated: 15 days ago

Promoted

Senior Data Engineer- Freelance

Quapton AI Solutionshosur, tamil nadu, in

Quapton AI Solutions is an innovative tech startup dedicated to providing advanced IT services within the field of artificial intelligence. We aim to empower businesses with tailored AI-driven solut...Show moreLast updated: 1 day ago