Principal Data Engineer

AutodeskBangalore, Bangalore (district)

2 days ago

Job description

# 25WD9141

Position Overview

We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.

Responsibilities

Technical Leadership

Define and drive the technical strategy for Batch Processing within ADP
Design and scale PySpark based distributed processing frameworks integrated with Airflow for orchestration
Champion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)
Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)
Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelines

Architecture & Delivery

Partner with architects to design multi-tenant, secure, and compliant (SOC2 / GDPR / CCPA) batch processing services

Define reference architectures and reusable frameworks for cross-domain processing workloads

Lead technical reviews, solution designs, and architectural forums across ADP

Guide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-service

Mentorship & Collaboration

Mentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teams

Partner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flows

Innovation & Impact

Drive modernization initiatives to migrate away from legacy tools

Pioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)

Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across Autodesk

Influence company-wide data engineering strategy by contributing thought leadership and whitepapers

Minimum Qualifications

8+ years of experience in software / data engineering, with at least 3 years in big data platform leadership roles

Expert in distributed data processing (Spark, PySpark, Ray, Flink)

Deep experience with workflow orchestration (Airflow, Dagster, Prefect)

Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS / Azure / GCP)

Proven track record in architecting secure, multi-tenant, and compliant data platforms

Skilled in SQL / NoSQL databases, Snowflake, Glue, and modern metadata / catalog tools

Strong problem-solving, communication, and cross-geo collaboration skills

Experience mentoring engineers, building strong technical culture, and influencing at scale

Preferred Qualifications

Exposure to AI / ML-driven data engineering (optimizers, anomaly detection, auto-scaling)

Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)

Familiarity with streaming + batch hybrid architectures (Kappa / Lambda)

#LI-KS2

Create a job alert for this search

Principal Data Engineer • Bangalore, Bangalore (district)

Related jobs

Promoted

Principal Engineer

Voya IndiaBangalore Urban, Karnataka, India

We are seeking a seasoned Principal Engineer with deep expertise in Java-based enterprise systems, cloud-native architectures, and technical leadership. This role will be pivotal in driving engineer...Show moreLast updated: 17 days ago

Promoted

Data Platform Engineer

Grizmo LabsBengaluru, Karnataka, India

Design, build, and maintain scalable data platforms and pipelines to support the needs of the business.Ensure data quality and accuracy by implementing data validation and testing processes.Partici...Show moreLast updated: 17 days ago

Promoted
New!

▷ (31 / 10 / 2025) Principal Data Engineer

PriceEasy AIBengaluru, Karnataka, India

About PriceEasy : Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insi...Show moreLast updated: 1 hour ago

Promoted

Principal Engineer- Data Interfaces

ABBBengaluru, Karnataka, India

This job is with ABB, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.At ABB, we help indu...Show moreLast updated: 4 days ago

Promoted

Principal AI Engineer - Data Platform

OutSystemsbangalore, India

There are NO limits to your career : come shape the future and be part of a truly unique global culture at OutSystems!.As a Principal AI Engineer at OutSystems, you will bridge research innovations ...Show moreLast updated: 30+ days ago

Promoted

Principal Data Engineer

NeemtreeBangalore

Description : Responsibilities : - Define and lead the data architecture vision and strategy, ensuring it supp...Show moreLast updated: 13 days ago

Promoted

Principal Software Engineer - Big Data Technologies

hirevedaBangalore

Key Responsibilities : - Lead technical initiatives and guide the team to develop innovative software solutions that address complex challenges.Build scalable, effici...Show moreLast updated: 30+ days ago

Promoted

Principal Data Engineer

PriceEasy AIBengaluru, Karnataka, India

Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insights through advan...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

Tata Consultancy ServicesBangalore Urban, Karnataka, India

TCS has been a great pioneer in feeding the fire of Techies like you.We are a global leader in the technology arena and there’s nothing that can stop us from growing together.Your role is of key im...Show moreLast updated: 30+ days ago

Promoted

AI & Data Principal Engineer

Talent SocioBangalore

About the Role : We are seeking a visionary AI & Data Principal Engineer to lead the architecture, development, and deployment of advanced AI and data-driven solutions.This...Show moreLast updated: 21 days ago

Promoted

Senior Data Engineer

USEReadyhosur, tamil nadu, in

Job Title : Senior Databricks Engineer.As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakeho...Show moreLast updated: 27 days ago

Promoted

Elanco - Principal Data Engineer

ElancoBangalore

Description : At Elanco (NYSE : ELAN) it all starts with animals!.As a global leader in animal health, we are dedicated to innovation and delivering products and se...Show moreLast updated: 2 days ago

Promoted

Principal Data Engineer

AutodeskBengaluru, Karnataka, India

We are seeking a highly experienced.Autodesk’s Analytics Data Platform (ADP).This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-o...Show moreLast updated: 6 days ago

Promoted

Principal AI Engineer [T500-19330]

Best Buy IndiaBengaluru, Karnataka, India

Lead operationalization of large-scale AI solutions, being the resident expert for all AI Engineering and AIOps challenges, partnering closely with AI Scientists. Lead by example, inspire and guide ...Show moreLast updated: 30+ days ago

Promoted

Principal Engineer-Devops

Dover IndiaGreater Bengaluru Area, India

Dover is a diversified global manufacturer with annual revenues of $8 billion.We deliver equipment and components, in specialty systems and support services through five major operating segments : E...Show moreLast updated: 30+ days ago

Promoted
New!

GCP Data Engineer

JRD Systemshosur, tamil nadu, in

Collaborate with data architects, analysts, and business teams to define and maintain.Roadnet) and modernized data sources. Partner with governance and infrastructure teams to establish.Automate and...Show moreLast updated: 19 hours ago

Promoted
New!

Principal Software Engineer - Data Platform

TalentiserBangalore

Job Description : Essential duties / responsibilities : - Lead technical initiatives and contribute as ...Show moreLast updated: 20 hours ago

Promoted

Principal Engineer

Wenger & WatsonBengaluru, Karnataka, India

Principal Engineer — Marketing & CRM.We’re seeking a Principal Engineer to drive the evolution of our cloud-native, SaaS-based marketing and CRM platform serving banks and credit unions across the ...Show moreLast updated: 4 days ago