Principal Data Engineer

AutodeskBengaluru, Karnataka, India

1 day ago

Job description

# 25WD9141

Position Overview

We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.

Responsibilities

Technical Leadership

Define and drive the technical strategy for Batch Processing within ADP
Design and scale PySpark based distributed processing frameworks integrated with Airflow for orchestration
Champion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)
Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)
Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelines

Architecture & Delivery

Partner with architects to design multi-tenant, secure, and compliant (SOC2 / GDPR / CCPA) batch processing services

Define reference architectures and reusable frameworks for cross-domain processing workloads

Lead technical reviews, solution designs, and architectural forums across ADP

Guide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-service

Mentorship & Collaboration

Mentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teams

Partner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flows

Innovation & Impact

Drive modernization initiatives to migrate away from legacy tools

Pioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)

Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across Autodesk

Influence company-wide data engineering strategy by contributing thought leadership and whitepapers

Minimum Qualifications

8+ years of experience in software / data engineering, with at least 3 years in big data platform leadership roles

Expert in distributed data processing (Spark, PySpark, Ray, Flink)

Deep experience with workflow orchestration (Airflow, Dagster, Prefect)

Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS / Azure / GCP)

Proven track record in architecting secure, multi-tenant, and compliant data platforms

Skilled in SQL / NoSQL databases, Snowflake, Glue, and modern metadata / catalog tools

Strong problem-solving, communication, and cross-geo collaboration skills

Experience mentoring engineers, building strong technical culture, and influencing at scale

Preferred Qualifications

Exposure to AI / ML-driven data engineering (optimizers, anomaly detection, auto-scaling)

Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)

Familiarity with streaming + batch hybrid architectures (Kappa / Lambda)

#LI-KS2

Create a job alert for this search

Principal Data Engineer • Bengaluru, Karnataka, India

Related jobs

Promoted

Principal Data Engineer

K&K Talents - IndiaBengaluru, IN

This position is with one of our.Title : Principal Data Engineer.Location : Bangalore, Bhopal, Chennai, Gurugram, Hyderabad, Jaipur, Pune. Employment Type : Full-time Permanent.GCP or AWS, Spark, Pyspa...Show moreLast updated: 11 days ago

Promoted
New!

Senior Data Engineer

AIQUhosur, tamil nadu, in

We are hiring for Senior Data Engineer to join one of our major clients based out of KSA.Employment Type : Contract – 12 months & extendable. The Senior Data Engineer plays a lead role in designing, ...Show moreLast updated: 18 hours ago

Promoted

Data Platform Engineer

Grizmo LabsBengaluru, Karnataka, India

Design, build, and maintain scalable data platforms and pipelines to support the needs of the business.Ensure data quality and accuracy by implementing data validation and testing processes.Partici...Show moreLast updated: 11 days ago

Promoted

Principal Engineer

FV Bankhosur, tamil nadu, in

FV Bank is a fully licensed and regulated U.With a focus on innovation, security, and compliance, FV Bank is Banking the Future by providing USD banking, digital asset custody services, money marke...Show moreLast updated: 1 day ago

Promoted

Principal Engineer

Hotel Traderhosur, tamil nadu, in

We're Hiring : Staff / Principal Engineer (Java) - Remote.Location : Remote | 🌍 Global Team | 💼 Experience : 8–12 years. Ready to build the future of hotel distribution at scale?.At Hotel Trader, we're...Show moreLast updated: 30+ days ago

Promoted
New!

Data Engineer

BrothersTechhosur, tamil nadu, in

Remote (Night Shift - Flexible).Must have excellent English written and verbal communication skills.The ideal candidate should be experienced in data engineering on. Snowflake Azure -ADF, Microsoft ...Show moreLast updated: 18 hours ago

Promoted

Principal Data Science Engineer - Data & Analytics [T500-20891]

Ferguson GCCBengaluru, Karnataka, India

Since 1953, Ferguson has been a source of quality supplies for a variety of industries.Together We Build Better infrastructure, better homes and better businesses. We exist to make our customers’ co...Show moreLast updated: 1 day ago

Promoted

Data Engineer

Manuh Technologieshosur, tamil nadu, in

S3, Glue, Redshift, EMR, Lambda).Develop automation scripts and tools using.Collaborate with data analysts, data scientists, and business stakeholders to ensure data availability and reliability.Tr...Show moreLast updated: 30+ days ago

Promoted

Principal Data Engineer

PriceEasy AIBengaluru, Karnataka, India

Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insights through advan...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

thinkbridgehosur, tamil nadu, in

We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Impetushosur, tamil nadu, in

Strong ability to write clean and efficient code.Good understanding of Spark SQL for distributed data processing.Experience with large datasets and structured data manipulation.Ability to write que...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

GEMRAJ TECHNOLOGIES LIMITEDhosur, tamil nadu, in

We are currently seeking a Data Engineer to join our technology team.In this role, you will work on building data-driven products that sit atop cloud-based data infrastructure.The ideal candidate i...Show moreLast updated: 3 days ago

Promoted

Data Engineer

V2Softhosur, tamil nadu, in

Experience to deal with large stream volumes.Implementation experience ( atleast couple of projects ) in MSK / Flink / Spark / Scala. Very good knowledge on atleast 3 out of the 4 technologies.Desirable e...Show moreLast updated: 11 days ago

Promoted

Data Engineer

VAANTECHhosur, tamil nadu, in

Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning. Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

USEReadyhosur, tamil nadu, in

Designation : Senior Data Engineer.As a part of the Data Warehouse Team, implement technology improvements to Enterprise Data Warehouse environment. In this role, you will be a key contributor to the...Show moreLast updated: 21 days ago

Promoted

Principal Engineer-Devops

Dover IndiaGreater Bengaluru Area, India

Dover is a diversified global manufacturer with annual revenues of $8 billion.We deliver equipment and components, in specialty systems and support services through five major operating segments : E...Show moreLast updated: 25 days ago

Promoted

Data Engineer

Talogyhosur, tamil nadu, in

This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago

Promoted

Lead Data Engineer

ITQube LTDhosur, tamil nadu, in

Looking for a Data Engineer to join our engineering will contribute directly to the design, automation, and optimization of our data processes, primarily developing solutions in Python within the A...Show moreLast updated: 21 days ago