Principal Data Pipeline Engineer
Innover DigitalBengaluru, Republic Of India, IN1 day ago
Job descriptionHands-on Migration and Development Pipeline Translation : Directly develop and migrate 20+ complex data pipelines from Azure Data Factory (ADF) into optimized Fabric Data Pipelines and Spark Notebooks (PySpark / Scala). Stored Procedure Conversion : Rewrite and optimize 30+ complex stored procedures into robust, scalable transformations using PySpark in Fabric Lakehouses or high-performance T-SQL in Fabric Data Warehouses. Data Modeling : Implement the designed data architecture (Medallion layers : Bronze, Silver, Gold) within OneLake, focusing on Delta Lake best practices and V-Order optimization for efficient querying. Large Data Handling : Execute the migration of 30+ TB of historical data efficiently and accurately, ensuring data integrity and reconciliation between the legacy and Fabric environments. Unit & Integration Testing : Develop and execute comprehensive unit and integration tests to validate the correctness and performance of migrated data pipelines and stored procedures. Performance Tuning : Work closely with the Technical Lead to identify and resolve data bottlenecks and optimize Spark code and Warehouse queries to meet latency requirements. Collaboration and DevOps Support CI / CD Adherence : Operate within the established DevOps pipeline, ensuring all code changes, notebooks, and object definitions are managed under Git version control and deployed via the automated CI / CD framework.
Required Qualifications
- Experience : 5+ years of dedicated experience in ETL / ELT development and data warehousing, with involvement in at least one major cloud data migration.
- Core Skills : Strong, demonstrable experience with the Azure Data Stack (ADF, Synapse) and hands-on development experience in Microsoft Fabric.
- Data Languages : Highly proficient in T-SQL for querying and procedural logic, and mandatory expertise in Python (PySpark) for data transformation.
- Data Modeling : Solid understanding of Kimball-style data warehousing, dimensional modeling, and modern Lakehouse concepts (Delta Lake, Parquet).
- Tools : Experience working with Git for source control and familiarity with the principles of CI / CD deployment.
Create a job alert for this search
Data Pipeline Engineer • Bengaluru, Republic Of India, IN
Related jobs
- Promoted
Backend and Data Pipeline Engineer
JRD Systemsbangalore, karnataka, in Job Role : Backend and Data Pipeline Engineer - Python.We’re investing in technology to develop new products that help our customers drive their growth and transformation agenda.These include new da...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer(Azure / AWS, Python / Pyspark, SQL)
Sail Analyticshosur, tamil nadu, in Architect, develop, test and maintain scalable data warehouses and data pipelines.Expertise in SQL, PySpark / Python and Azure(ADB, ADF) or AWS(Glue, Lambda, Redshift).
Bachelor's degree or equivalent...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Recrohosur, tamil nadu, in Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake.
AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago
- Promoted
Python Data Engineer
Dexian Indiahosur, tamil nadu, in Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights.
Constructing infrastructure for efficient ETL processes from vari...Show moreLast updated: 1 day ago
- Promoted
Senior Data Engineer (AWS / Databricks)
Accoladehosur, tamil nadu, in The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 30+ days ago
- Promoted
Principal Data Engineer
PriceEasy AIBengaluru, Karnataka, India Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insights through advan...Show moreLast updated: 30+ days ago
- Promoted
Azure Data Engineer
LTIMindtreehosur, tamil nadu, in Role Senior Data Engineer 8 years of experience.Build reusable utilities templates and automation pipelines.Design scalable data engineering frameworks standards and best practices.Provide architec...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Digitalzonehosur, tamil nadu, in As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics.
Develop and maintain scalable ETL / ELT pipelines using Py...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
Veraxionhosur, tamil nadu, in Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions.
Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 12 days ago
- Promoted
Data Engineer
VAANTECHhosur, tamil nadu, in Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning.
Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer II
ClearDemandhosur, tamil nadu, in Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
You'll play a crucial role in driving the evolution and efficiency of our...Show moreLast updated: 1 day ago
- Promoted
Freelance Data Engineer
upGradhosur, tamil nadu, in We are seeking a highly skilled and motivated.The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a ...Show moreLast updated: 20 days ago
- Promoted
Principal Data Engineer
AutodeskBengaluru, Karnataka, India We are seeking a highly experienced.Autodesk’s Analytics Data Platform (ADP).This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-o...Show moreLast updated: 12 days ago
- Promoted
- New!
Principal Data Engineer
CodeMyMobileBangalore, IN Experience Required - 7 to 10 Years.Are you a Data Engineer who cares about clean engineering, autonomy, and solving real data challenges? If this sounds like you, we’d love to connect!.Email your ...Show moreLast updated: 20 hours ago
- Promoted
- New!
Senior Data Engineer – ETL & Pipeline Development (7 to 12 yrs)
AIMLEAPhosur, tamil nadu, in Senior Data Engineer – ETL & Pipeline Development.Remote (Work from Home) / Bangalore / India.Tech / MCA / Computer Science / IT.
IT / Data / AI / LegalTech / Enterprise Solutions.Pandas, Airflow, o...Show moreLast updated: 15 hours ago
- Promoted
GCP Data Engineer
JRD Systemshosur, tamil nadu, in Collaborate with data architects, analysts, and business teams to define and maintain.Roadnet) and modernized data sources.
Partner with governance and infrastructure teams to establish.Automate and...Show moreLast updated: 6 days ago
- Promoted
Lead Data Engineer
Cimpresshosur, tamil nadu, in Our Team : Enterprise Business Solutions.Vista’s Enterprise Business Solutions (EBS) domain is working to make our company one of the most data-driven organizations to support Finance, Supply Chain,...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
LTIMindtreehosur, tamil nadu, in We are seeking a skilled Data Engineer with hands-on experience in.The ideal candidate will be responsible for developing scalable data pipelines, transforming and ingesting large-scale data, and e...Show moreLast updated: 30+ days ago