No longer accepting applications

Senior Data Engineer

Insight GlobalHyderabad, Telangana, India

6 days ago

Job description

Summary

The Senior Data Engineer is responsible for building and optimizing ETL / ELT pipelines that process terabytes of data daily across 186 data assets, implementing BigQuery datasets with enterprise-scale performance optimization, and creating the data quality monitoring and transparency dashboards that enable data owner self-service.

Required Qualifications

Google Cloud Platform Data Engineering

5+ years of data engineering experience with at least 2+ years focused on Google Cloud Platform
Strong proficiency with BigQuery including :
Advanced SQL for analytical queries (window functions, CTEs, complex joins)
Partitioning and clustering strategies for performance optimization
Materialized views, authorized views, and query optimization techniques
Cost optimization through efficient query design and storage management
Understanding of BigQuery architecture (slot allocation, shuffle operations, distributed execution)
Hands-on experience with Google Cloud Dataflow and Apache Beam :
Pipeline development in Python or Java
Batch and streaming data processing patterns
Performance tuning and resource optimization
Error handling and pipeline monitoring
Proficiency with Cloud Composer (Apache Airflow) :
DAG development and dependency management
Airflow operators (BigQueryOperator, DataflowOperator, custom operators)
Workflow orchestration for complex multi-step processes
Monitoring and troubleshooting failed workflows

ETL / ELT & Data Integration

Strong experience building production-grade ETL / ELT pipelines processing terabyte-scale data

Knowledge of data integration patterns (full refresh, incremental load, change data capture)

Experience with data transformation techniques (normalization, denormalization, aggregation)

Understanding of data quality frameworks and validation strategies

Proficiency with schema evolution handling changes without breaking downstream systems

Experience with data lineage tracking from source to consumption

SQL & Database Technologies

Expert-level SQL skills including advanced analytics functions and query optimization

Understanding of database performance tuning (indexing, partitioning, query plans)

Experience with relational databases (PostgreSQL, MySQL, SQL Server) for source system integration

Familiarity with NoSQL databases (Firestore, Bigtable) for specialized use cases

Knowledge of data warehousing concepts (fact tables, dimension tables, slowly changing dimensions)

Programming & Scripting

Strong Python proficiency for data pipeline development and scripting

Experience with Apache Beam SDK for Dataflow pipeline development

Proficiency with Pandas, NumPy for data manipulation and analysis

Understanding of object-oriented programming and software engineering best practices

Experience with Git for version control and collaborative development

Basic Shell scripting for automation and operational tasks

Data Security & Compliance

Understanding of row-level security implementation patterns in BigQuery

Experience with PHI / PII data handling and healthcare compliance requirements (HIPAA preferred)

Knowledge of data masking and de-identification techniques

Understanding of audit logging and compliance reporting requirements

Familiarity with least privilege principles and data access controls

Preferred Qualifications

Google Cloud Professional Data Engineer certification

Healthcare industry experience with understanding of clinical and administrative data

Experience with Google Cloud Storage lifecycle policies and storage class optimization

Knowledge of Cloud Spanner for transactional workloads

Familiarity with Cloud DLP API for automated data classification

Experience with dbt (data build tool) for analytics engineering

Understanding of data mesh or data fabric architectural patterns

Background in DevOps practices and CI / CD for data pipelines

Experience with Terraform for infrastructure as code

Knowledge of data visualization tools (Looker, Tableau, Power BI)

Familiarity with machine learning workflows on GCP (Vertex AI)

Experience with Docker and Kubernetes (GKE) for containerized workloads

Create a job alert for this search

Senior Data Engineer • Hyderabad, Telangana, India

Related jobs

Promoted

Senior Data Engineer

ValueLabsHyderabad, Telangana, India

As a Senior Data Engineer, you will be the lead architect and builder of the data backbone for our entire next-generation digital asset platform. Embedded within our "Agentic AI & Automation" squad,...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Prudent Technologies and Consulting, Inc.Hyderabad, Telangana, India

Seeking an experienced Senior Data Engineer with over 9 years of industry experience to architect, build, and manage advanced data pipelines and platforms in a cloud environment.The ideal candidate...Show moreLast updated: 30+ days ago

Promoted
New!

Senior Data Engineer - AWS & Python

Egenhyderabad, telangana, in

Design, develop, and maintain ETL / ELT data pipelines using Python and AWS native services (Glue, Lambda, EMR, Step Functions, etc. Build and manage data lakes and data warehouses using Amazon S3, Re...Show moreLast updated: 15 hours ago

Promoted

Senior Data Engineer

Saras AnalyticsHyderabad, Telangana, India

About Saras Analytics : We are an ecommerce focused end to end data analytics firm assisting enterprises & brands in data driven decision making to maximize business value.Our suite of work spans ex...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

ExperianHyderabad, Telangana, India

Experian Consumer Services – Careers That Define “What’s the Next Big (Data) Thing” for Consumers?.What could be more exciting – personally and professionally – than being part of a “disruptive” bu...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

SGS & Cohyderabad, telangana, in

Position Title : Senior Data Engineer.Experience Required : 8 to 12 Years.We are looking for a highly skilled and experienced Data Engineer with strong expertise in. The ideal candidate will play a ke...Show moreLast updated: 16 days ago

Promoted

Senior AWS Data Engineer

CYAN360Hyderabad, IN

Position : Senior AWS Data Engineer.Work Timings : 2 : 30 PM to 11 : 30 PM IST.Need someone who can join immediately or in 15 days • • •. Design, develop, and deploy end-to-end data pipelines on AWS cloud in...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

DonyatiHyderabad, IN

We are seeking a highly skilled Senior Data Engineer to join our team in building a modern data platform on AWS.You will play a key role in transitioning from legacy systems to a scalable, cloud-na...Show moreLast updated: 1 day ago

Promoted

Senior Data Engineer

Multiscale Technologies IndiaHyderabad, Telangana, India

Data Engineer, Forward Deployed Applications Location : Hyderabad, India (on-site at client location) Work Site : On-site Travel : 0–10% Role Type : Individual Contributor Employment Type : Full-ti...Show moreLast updated: 12 days ago

Promoted

Senior Data Engineer (Azure Databricks)

StratusHyderabad, Republic Of India, IN

Lead a team of data engineers to develop data analytic solutions such as data warehouse and data marts on Azure Databricks. Experience with Agile development methodologies and tools, such as Jira, t...Show moreLast updated: 9 days ago

Promoted

Senior Data Solutions Engineer

Accordion IndiaHyderabad, Republic Of India, IN

Accordion is a global private equity-focused financial consulting firm specializing in driving value creation through services rooted in Data & Analytics and powered by technology.Accordion works a...Show moreLast updated: 12 days ago

Promoted

Senior Data Engineer

EXLHyderabad, Telangana, India

Job Summary : We are seeking an experienced Data Engineer with strong expertise in AWS cloud services and Databricks to design and implement scalable, high-performance data pipelines.Key Responsibi...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

Adaptive Technology InsightsHyderabad, India

We are looking for an experienced Data Engineer with strong expertise in Google Cloud Platform (GCP) and BigQuery to join our growing data team. You will be responsible for designing, building...Show moreLast updated: 1 day ago

Promoted

Senior Data Platform Engineer

ValueLabsHyderabad, Republic Of India, IN

Promoted

Senior Data Engineer

SAIVA AIHyderabad, IN

We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 30+ days ago

Promoted

Senior Data Solutions Engineer

ExperianHyderabad, Republic Of India, IN

Promoted

Senior Data Engineer

EverestDX IncHyderabad, Telangana, India

Everest DX – We are a Digital Platform Services company, headquartered in Stamford.Our Platform / Solution includes Orchestration, Intelligent operations with BOTs’, AI-powered analytics for Enterpri...Show moreLast updated: 2 days ago

Promoted

Senior Data Engineer

Quadrant Technologieshyderabad, telangana, in

Senior Data Engineer (10+ Years).Location : Chennai, Bengaluru, Hyderabad.Drive end-to-end cloud data migration, architecture design (Fabric, Synapse, Databricks), and solution delivery.Build and op...Show moreLast updated: 30+ days ago