Principal Data Pipeline EngineerInnover Digital • Bengaluru, Republic Of India, IN
No longer accepting applications
Principal Data Pipeline Engineer
Innover Digital • Bengaluru, Republic Of India, IN
22 days ago
Job descriptionHands-on Migration and Development Pipeline Translation : Directly develop and migrate 20+ complex data pipelines from Azure Data Factory (ADF) into optimized Fabric Data Pipelines and Spark Notebooks (PySpark / Scala). Stored Procedure Conversion : Rewrite and optimize 30+ complex stored procedures into robust, scalable transformations using PySpark in Fabric Lakehouses or high-performance T-SQL in Fabric Data Warehouses. Data Modeling : Implement the designed data architecture (Medallion layers : Bronze, Silver, Gold) within OneLake, focusing on Delta Lake best practices and V-Order optimization for efficient querying. Large Data Handling : Execute the migration of 30+ TB of historical data efficiently and accurately, ensuring data integrity and reconciliation between the legacy and Fabric environments. Unit & Integration Testing : Develop and execute comprehensive unit and integration tests to validate the correctness and performance of migrated data pipelines and stored procedures. Performance Tuning : Work closely with the Technical Lead to identify and resolve data bottlenecks and optimize Spark code and Warehouse queries to meet latency requirements. Collaboration and DevOps Support CI / CD Adherence : Operate within the established DevOps pipeline, ensuring all code changes, notebooks, and object definitions are managed under Git version control and deployed via the automated CI / CD framework.
Required Qualifications
- Experience : 5+ years of dedicated experience in ETL / ELT development and data warehousing, with involvement in at least one major cloud data migration.
- Core Skills : Strong, demonstrable experience with the Azure Data Stack (ADF, Synapse) and hands-on development experience in Microsoft Fabric.
- Data Languages : Highly proficient in T-SQL for querying and procedural logic, and mandatory expertise in Python (PySpark) for data transformation.
- Data Modeling : Solid understanding of Kimball-style data warehousing, dimensional modeling, and modern Lakehouse concepts (Delta Lake, Parquet).
- Tools : Experience working with Git for source control and familiarity with the principles of CI / CD deployment.
Create a job alert for this search
Data Pipeline Engineer • Bengaluru, Republic Of India, IN
Related jobs
Principal Data Pipeline Engineer
Andor Tech • Bengaluru, Republic Of India, IN
If you're passionate about big data technologies and thrive in solving complex problems, we want to hear from you!.Deployment to a Product-Based Retail MNC.
Work on cutting-edge big data solutions f...Show more
Last updated: 21 days ago • Promoted
Lead Data Pipeline Engineer
Arcana • Bengaluru, Republic Of India, IN
Join our dynamic team at the forefront of cutting-edge technology as we seek a seasoned Senior Data Engineer.Embark on a journey where your deep-rooted expertise in computer science fundamentals, a...Show more
Last updated: 4 days ago • Promoted
Data Pipeline Engineer
Luxoft • Bengaluru, Republic Of India, IN
This project is part of a strategic initiative to migrate legacy on-premises data systems to a modern, scalable cloud-based data platform using Snowflake on Azure.
The goal is to enhance data access...Show more
Last updated: 30+ days ago • Promoted
Senior Data Pipeline Engineer
AppsForBharat • Bengaluru, Republic Of India, IN
We are looking for a Senior Data Engineer to lead the design and development of a next-generation data platform that powers real-time and batch applications at scale.
This platform supports business...Show more
Last updated: 22 days ago • Promoted
Principal Data Pipeline Engineer
Incept Labs • Bengaluru, Republic Of India, IN
Incept Labs is an AI research lab based in San Francisco, California.Our small team of scientists, engineers, and builders is passionate about developing domain-specific, next-generation AI solutio...Show more
Last updated: 30+ days ago • Promoted
Senior Data Pipeline Engineer
Cargill • Bengaluru, Republic Of India, IN
The Senior Professional, Data Engineering designs, builds, and maintains complex data systems that enable data analysis and reporting.
This role includes a focus on Treasury-specific data solutions,...Show more
Last updated: 13 days ago • Promoted
Principal Data Pipeline Engineer
Ascendion • Bengaluru, Republic Of India, IN
We’re Hiring – GCP BigQuery Developers!.Are you passionate about building scalable data solutions and working with cutting-edge cloud technologies?.
Bangalore / Hyderabad / Chennai / Pune.Immediate to 15 ...Show more
Last updated: 19 days ago • Promoted
Senior Data Pipeline Engineer
Brillio • Bengaluru, Republic Of India, IN
Brillio is a global digital technology company focused on delivering innovative solutions in data, AI, and cloud.With a strong presence across industries like banking, healthcare, and telecom, Bril...Show more
Last updated: 30+ days ago • Promoted
Principal Data Engineer
PriceEasy AI • Bengaluru, Karnataka, India
Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insights through advan...Show more
Last updated: 30+ days ago • Promoted
Lead Data Pipeline Engineer
ACL Digital • Bengaluru, Republic Of India, IN
Tech Lead design, build, and maintain the infrastructure for an organization's data, ensuring it is reliable, accessible, and optimized for use by data scientists and analysts.Key responsibilities ...Show more
Last updated: 20 days ago • Promoted
Lead Data Pipeline Engineer
Cimpress India • Bengaluru, Republic Of India, IN
NASDAQ : CMPR) is the world leader in mass customisation.Our unmatched technology, production, and supply chain operations allow us to offer products that can be personalised by an individual custom...Show more
Last updated: 30+ days ago • Promoted
Lead Data Pipeline Engineer
TEKsystems Global Services in India • Bengaluru, Republic Of India, IN
Develop and manage data pipelines using ADF and Azure data bricks.Expert knowledge in data warehousing, data modeling and ETL.
Good to have datastage exposure.Expertise in on-prem to azure cloud mig...Show more
Last updated: 4 days ago • Promoted
Data Pipeline Engineer
Mareana • Bengaluru, Republic Of India, IN
Mareana is looking for experienced Data Engineer with a strong development background using, RDMS, graph database and NoSQL platforms.
This role is responsible for developing and enhancing Mareana’s...Show more
Last updated: 30+ days ago • Promoted
Data Pipeline Engineer
L&T Technology Services • Bengaluru, Republic Of India, IN
PySpark, SQL, AWS and good to have Pandas.Show more
Last updated: 7 days ago • Promoted
Lead Data Pipeline Engineer
Zluri • Bengaluru, Republic Of India, IN
Job Location : Whitefield, Bangalore.Reporting to : Engineering Manager.Zluri is a next-gen Identity Governance and Administration (IGA) platform that enables IT and security teams to discover iden...Show more
Last updated: 19 days ago • Promoted
Senior Data Pipeline Engineer
Quess IT Staffing • Bengaluru, Republic Of India, IN
We are looking for a highly skilled.The ideal candidate should have deep expertise in.Snowflake, PySpark, SQL, DBT, Airflow.Show more
Last updated: 7 days ago • Promoted
Principal Data Engineer
Tata Consultancy Services • Bengaluru, Karnataka, India
Skill : Principal Data Engineer.Possesses hands-on technical experience working in AWS.The individual should possess a robust set of technical and soft skills and be an excellent AWS Data Engineer ...Show more
Last updated: 9 days ago • Promoted
Data Pipeline Engineer
Emids • Bengaluru, Republic Of India, IN
We are looking for a highly skilled Data Engineer with strong expertise in AWS, Databricks, PySpark, and Airflow to join our growing Data Engineering team.
The ideal candidate will be responsible fo...Show more
Last updated: 1 day ago • Promoted