Principal Data Engineer

AutodeskBangalore (division)

4 days ago

Job description

# 25WD9141

Position Overview

We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.

Responsibilities

Technical Leadership

Define and drive the technical strategy for Batch Processing within ADP
Design and scale PySpark based distributed processing frameworks integrated with Airflow for orchestration
Champion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)
Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)
Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelines

Architecture & Delivery

Partner with architects to design multi-tenant, secure, and compliant (SOC2 / GDPR / CCPA) batch processing services

Define reference architectures and reusable frameworks for cross-domain processing workloads

Lead technical reviews, solution designs, and architectural forums across ADP

Guide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-service

Mentorship & Collaboration

Mentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teams

Partner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flows

Innovation & Impact

Drive modernization initiatives to migrate away from legacy tools

Pioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)

Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across Autodesk

Influence company-wide data engineering strategy by contributing thought leadership and whitepapers

Minimum Qualifications

8+ years of experience in software / data engineering, with at least 3 years in big data platform leadership roles

Expert in distributed data processing (Spark, PySpark, Ray, Flink)

Deep experience with workflow orchestration (Airflow, Dagster, Prefect)

Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS / Azure / GCP)

Proven track record in architecting secure, multi-tenant, and compliant data platforms

Skilled in SQL / NoSQL databases, Snowflake, Glue, and modern metadata / catalog tools

Strong problem-solving, communication, and cross-geo collaboration skills

Experience mentoring engineers, building strong technical culture, and influencing at scale

Preferred Qualifications

Exposure to AI / ML-driven data engineering (optimizers, anomaly detection, auto-scaling)

Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)

Familiarity with streaming + batch hybrid architectures (Kappa / Lambda)

#LI-KS2

Create a job alert for this search

Principal Data Engineer • Bangalore (division)

Related jobs

Promoted

Principal Data Engineer

PriceEasy AIBangalore (division)

Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insights through advan...Show moreLast updated: 3 days ago

Promoted

Data Engineer

Randstad EnterpriseTumkur, IN

Shift Timing : 2 : 00 Pm - 11 : 00 Pm.Experience : 2- 4 years relevant Experience only ( this is a Junior position with us ). GCP - 2 years minimum working Experience.Worked with global stakeholders.Ran...Show moreLast updated: 30+ days ago

Promoted

Freelance Data Engineer

upGradTumkur, IN

We are seeking a highly skilled and motivated.The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a ...Show moreLast updated: 17 days ago

Promoted

Senior AWS Data Engineer

CYAN360Tumkur, IN

Position : Senior AWS Data Engineer.Work Timings : 2 : 30 PM to 11 : 30 PM IST.Need someone who can join immediately or in 15 days • • •. Design, develop, and deploy end-to-end data pipelines on AWS cloud in...Show moreLast updated: 30+ days ago

Promoted

Principal Engineer

Hotel TraderTumkur, IN

We're Hiring : Staff / Principal Engineer (Java) - Remote.Location : Remote | 🌍 Global Team | 💼 Experience : 8–12 years. Ready to build the future of hotel distribution at scale?.At Hotel Trader, we're...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

IntraEdgeTumkur, IN

The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both interna...Show moreLast updated: 30+ days ago

Promoted

AWS Data Engineer

TerraGiGTumkur, IN

Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR. Writing reusable, testable, and efficient code.Integration of data storage ...Show moreLast updated: 8 days ago

Promoted

Data Engineer

RevXTumkur, IN

RevX helps app businesses acquire and reengage users via programmatic to retain, monetize, and accelerate revenue.We're all about taking your app businesses to a new growth level.We rely on data sc...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Response InformaticsTumkur, IN

We’re Hiring | Data Engineering Experts (Hyderabad / Remote).We’re looking for passionate and experienced.Data Engineering professionals. If you love building scalable data pipelines, optimizing clo...Show moreLast updated: 18 days ago

Promoted

Data Engineer

VeraxionTumkur, IN

Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions. Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 8 days ago

Promoted

Senior Data Platform Engineer

Black Dog LabsTumkur, IN

Remote (collaboration across time zones), India or LATAM preferred.Proficient English communication.Data Engineering / Backend Engineering / DevOps. We’re looking for a hands-on Senior Data Platform...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

AAA GlobalTumkur, IN

Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team. In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 18 days ago

Promoted

Senior Data Engineer

SAIVA AITumkur, IN

We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer

HealthyMe AITumkur, IN

We are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data.You will report to the Head of AI and Engineering and be responsible for designi...Show moreLast updated: 27 days ago

Promoted

Python AWS Data Engineer

Digitrix Software LLPTumkur, IN

Mandatory skills - python , pyspark, who can write codes , any cloud exp - aws / gcp / azure.Python, AWS Python (core language skill) Backend, Pandas, PySpark (DataFrame API), interacting with AW...Show moreLast updated: 30+ days ago

Promoted

GCP Data Engineer

EXLTumkur, IN

Must Have- GCP Data Engineer with Banking / Finance Institutions Experience.Google Cloud Platform (GCP) Engineers.This solution will integrate data from diverse sources including.SAS systems, Excel ...Show moreLast updated: 18 days ago

Promoted

Data Engineer_Immediate Joiners

VXI Global SolutionsTumkur, IN

VXI Global Solutions is a BPO leader in customer service, customer experience, and digital solutions.Founded in 1998, the company has 40,000 employees in more than 40 locations in North America, As...Show moreLast updated: 18 days ago

Promoted

AI / ML & Data Engineer

Mindfire SolutionsTumkur, IN

We are looking for an experienced AI / ML & Data Engineer to design, develop, and deploy scalable machine learning models and data infrastructure on AWS. You will work closely with cross-functional te...Show moreLast updated: 10 days ago