No longer accepting applications

Senior Data Engineer

Insight Globalsrikakulam, andhra pradesh, in

9 days ago

Job description

Summary

The Senior Data Engineer is responsible for building and optimizing ETL / ELT pipelines that process terabytes of data daily across 186 data assets, implementing BigQuery datasets with enterprise-scale performance optimization, and creating the data quality monitoring and transparency dashboards that enable data owner self-service.

Required Qualifications

Google Cloud Platform Data Engineering

5+ years of data engineering experience with at least 2+ years focused on Google Cloud Platform
Strong proficiency with BigQuery including :
Advanced SQL for analytical queries (window functions, CTEs, complex joins)
Partitioning and clustering strategies for performance optimization
Materialized views, authorized views, and query optimization techniques
Cost optimization through efficient query design and storage management
Understanding of BigQuery architecture (slot allocation, shuffle operations, distributed execution)
Hands-on experience with Google Cloud Dataflow and Apache Beam :
Pipeline development in Python or Java
Batch and streaming data processing patterns
Performance tuning and resource optimization
Error handling and pipeline monitoring
Proficiency with Cloud Composer (Apache Airflow) :
DAG development and dependency management
Airflow operators (BigQueryOperator, DataflowOperator, custom operators)
Workflow orchestration for complex multi-step processes
Monitoring and troubleshooting failed workflows

ETL / ELT & Data Integration

Strong experience building production-grade ETL / ELT pipelines processing terabyte-scale data

Knowledge of data integration patterns (full refresh, incremental load, change data capture)

Experience with data transformation techniques (normalization, denormalization, aggregation)

Understanding of data quality frameworks and validation strategies

Proficiency with schema evolution handling changes without breaking downstream systems

Experience with data lineage tracking from source to consumption

SQL & Database Technologies

Expert-level SQL skills including advanced analytics functions and query optimization

Understanding of database performance tuning (indexing, partitioning, query plans)

Experience with relational databases (PostgreSQL, MySQL, SQL Server) for source system integration

Familiarity with NoSQL databases (Firestore, Bigtable) for specialized use cases

Knowledge of data warehousing concepts (fact tables, dimension tables, slowly changing dimensions)

Programming & Scripting

Strong Python proficiency for data pipeline development and scripting

Experience with Apache Beam SDK for Dataflow pipeline development

Proficiency with Pandas, NumPy for data manipulation and analysis

Understanding of object-oriented programming and software engineering best practices

Experience with Git for version control and collaborative development

Basic Shell scripting for automation and operational tasks

Data Security & Compliance

Understanding of row-level security implementation patterns in BigQuery

Experience with PHI / PII data handling and healthcare compliance requirements (HIPAA preferred)

Knowledge of data masking and de-identification techniques

Understanding of audit logging and compliance reporting requirements

Familiarity with least privilege principles and data access controls

Preferred Qualifications

Google Cloud Professional Data Engineer certification

Healthcare industry experience with understanding of clinical and administrative data

Experience with Google Cloud Storage lifecycle policies and storage class optimization

Knowledge of Cloud Spanner for transactional workloads

Familiarity with Cloud DLP API for automated data classification

Experience with dbt (data build tool) for analytics engineering

Understanding of data mesh or data fabric architectural patterns

Background in DevOps practices and CI / CD for data pipelines

Experience with Terraform for infrastructure as code

Knowledge of data visualization tools (Looker, Tableau, Power BI)

Familiarity with machine learning workflows on GCP (Vertex AI)

Experience with Docker and Kubernetes (GKE) for containerized workloads

Create a job alert for this search

Senior Data Engineer • srikakulam, andhra pradesh, in

Related jobs

Promoted

Senior DataOps Engineer (AWS)

MSBC Groupsrikakulam, India

Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services.Lead the E-Comms data pipeline within Compass’s Application Simplification workstream : design, ...Show moreLast updated: 23 hours ago

Promoted

Data Engineer

BayOne Solutionssrikakulam, andhra pradesh, in

We are seeking a highly experienced Data Engineer to join our MarTech team and play a pivotal role in driving innovation within our microservices architecture, with a strong emphasis on data engine...Show moreLast updated: 3 days ago

Promoted

Senior Data Engineer

SGS & Cosrikakulam, andhra pradesh, in

Position Title : Senior Data Engineer.Experience Required : 8 to 12 Years.We are looking for a highly skilled and experienced Data Engineer with strong expertise in. The ideal candidate will play a ke...Show moreLast updated: 17 days ago

Promoted

Databricks Engineer

TTC Groupsrikakulam, andhra pradesh, in

We are seeking a Mid-Level Databricks Engineer with strong data engineering fundamentals and hands-on experience building scalable data pipelines on the Databricks platform.The ideal candidate will...Show moreLast updated: 1 day ago

Promoted

Azure Data Engineer

Tata Consultancy Servicessrikakulam, andhra pradesh, in

TCS is Hiring for Azure Data Engineer.Primary : Azure Data Factory (ADF), Azure Databricks (ADB).Secondary : SSIS (SQL Server Integration Services). Strong expertise in Azure Data Engineering & ETL Co...Show moreLast updated: 30+ days ago

Promoted
New!

AWS Data Engineer

COVET IT INCsrikakulam, andhra pradesh, in

Please go through the below requirements and let me know your interest and forward your resume along with your contact information to raja@covetitinc. We are seeking a highly skilled AWS Data Engine...Show moreLast updated: 5 hours ago

Promoted

Senior Data Engineer

American Inferencesrikakulam, India

Please apply directly through LinkedIn for this position.We kindly request that candidates refrain from contacting company officials via email or messages regarding this role.We are an AI and Data ...Show moreLast updated: 14 days ago

Promoted

Senior Data Engineer

Masteksrikakulam, India

We are looking for a Senior Data Engineer with good exposure to Cloud Services in AWS(IAM, RDS, Lambda, DMS) along with Snowflake / Postgresql, ETL and Streaming Services like Kafka.Show moreLast updated: 15 days ago

Promoted
New!

Azure Data Engineer

Capgemini Inventsrikakulam, India

At Capgemini Invent, we believe difference drives change.As inventive transformation consultants, we blend our strategic, creative and scientific capabilities, collaborating closely with clients to...Show moreLast updated: 22 hours ago

Promoted
New!

Senior Data Engineer / Architect - Snowflake

Aezion, IncChicacole, Republic Of India, IN

We are seeking a highly skilled Snowflake Data Engineer with expertise in DBT, Airbyte, and Airflow, combined with strong experience in Kimball dimensional modeling. The ideal candidate will design ...Show moreLast updated: 22 hours ago

Promoted

Data Engineer

Alp Consulting Ltd.Srikakulam, Andhra Pradesh, India

Key Responsibilities • Architect and maintain our Amazon Redshift Serverless data warehouse.Design and implement ETL pipelines from operational Redshift to staging (DSA), landing (DLA), and TDW lay...Show moreLast updated: 30+ days ago

Promoted
New!

Senior Data Engineer

Mindlancesrikakulam, India

We’re looking for a strong, hands-on Sr Data Engineer who can independently drive client conversations, collaborate with SMEs, and deliver high-quality data solutions in a cloud-native environment....Show moreLast updated: 12 hours ago

Promoted

Senior Azure Data Engineer

Tata Consultancy ServicesSrikakulam, Andhra Pradesh, India

Greetings from TCS! Job Title : Senior Azure Data Engineer Required Skillset : Azure data factory, data bricks, data lake, automation and performance optimization of ETL. Location : Gurugram Experience...Show moreLast updated: 1 day ago

Promoted

Sr. Data Engineer

HNR Techsrikakulam, andhra pradesh, in

We are seeking an experienced Senior Data Engineer to design, build, and optimize our modern data ecosystem.The ideal candidate will have deep expertise in data architecture, pipeline orchestration...Show moreLast updated: 10 days ago

Promoted

(Senior) Azure Data Engineer - Full remote - contractor in USD

All European Careerssrikakulam, andhra pradesh, in

Remote

For an international project in Chennai, we are urgently looking for a Full Remote Senior Azure Data Engineer, who will build data pipeline for enterprise search applications using ADF and Databric...Show moreLast updated: 3 days ago

Promoted

Data Engineer

MyData Insights Pvt LtdSrikakulam, Andhra Pradesh, India

Role : Data Engineer Experience : 6–12 years Location : Remote Engagement : Contract (4–6 Months) Work Type : Remote About the Role We are seeking an experienced Data Engineer to support data mo...Show moreLast updated: 1 day ago

Promoted

Senior Python Data Engineer

iVoyantsrikakulam, andhra pradesh, in

Join a dynamic engineering team working on a high-impact tax reporting platform for the 2025 fiscal season.The core goal is to modernize and significantly accelerate the generation of Excel-based r...Show moreLast updated: 4 days ago

Promoted

Senior GenAI Engineer

Mitra AIsrikakulam, andhra pradesh, in

AI System Design & Development : .Architect, develop, and deploy large-scale Generative AI, LLM-based systems, including intelligent agents and automation workflows. LLM Integration & Optimization : .In...Show moreLast updated: 1 day ago