Principal Data Engineer

AutodeskBengaluru, Karnataka, India

18 days ago

Job description

#25WD9141

Position Overview

We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform, which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.

Responsibilities

Technical Leadership

Define and drive the technical strategy for Batch Processing within ADP
Design and scale PySpark based distributed processing frameworks integrated with Airflow for orchestration
Champion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)
Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)
Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelines

Architecture & Delivery

Partner with architects to design multi-tenant, secure, and compliant (SOC2 / GDPR / CCPA) batch processing services

Define reference architectures and reusable frameworks for cross-domain processing workloads

Lead technical reviews, solution designs, and architectural forums across ADP

Guide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-service

Mentorship & Collaboration

Mentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teams

Partner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flows

Innovation & Impact

Drive modernization initiatives to migrate away from legacy tools

Pioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)

Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across Autodesk

Influence company-wide data engineering strategy by contributing thought leadership and whitepapers

Minimum Qualifications

8+ years of experience in software / data engineering, with at least 3 years in big data platform leadership roles

Expert in distributed data processing (Spark, PySpark, Ray, Flink)

Deep experience with workflow orchestration (Airflow, Dagster, Prefect)

Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS / Azure / GCP)

Proven track record in architecting secure, multi-tenant, and compliant data platforms

Skilled in SQL / NoSQL databases, Snowflake, Glue, and modern metadata / catalog tools

Strong problem-solving, communication, and cross-geo collaboration skills

Experience mentoring engineers, building strong technical culture, and influencing at scale

Preferred Qualifications

Exposure to AI / ML-driven data engineering (optimizers, anomaly detection, auto-scaling)

Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)

Familiarity with streaming + batch hybrid architectures (Kappa / Lambda)

#LI-KS2

Create a job alert for this search

Principal Data Engineer • Bengaluru, Karnataka, India

Related jobs

Promoted

Principal Data Engineer

CodeMyMobileBengaluru, IN

Experience Required - 7 to 10 Years.Are you a Data Engineer who cares about clean engineering, autonomy, and solving real data challenges? If this sounds like you, we’d love to connect!.Email your ...Show moreLast updated: 7 days ago

Promoted

Principal Engineer

CoffeeBeansbangalore, karnataka, in

CoffeeBeans Consulting is a tech consulting firm focused on making organizations.We are looking for a Principal Engineer who can design and implement complex solutions that are scalable, future-pro...Show moreLast updated: 30+ days ago

Promoted

Principal Engineer

FV Bankhosur, tamil nadu, in

FV Bank is a fully licensed and regulated U.With a focus on innovation, security, and compliance, FV Bank is Banking the Future by providing USD banking, digital asset custody services, money marke...Show moreLast updated: 18 days ago

Promoted

Principal Engineer

Hotel Traderhosur, tamil nadu, in

We're Hiring : Staff / Principal Engineer (Java) - Remote.Location : Remote | 🌍 Global Team | 💼 Experience : 8–12 years. Ready to build the future of hotel distribution at scale?.At Hotel Trader, we're...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Recrohosur, tamil nadu, in

Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake. AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago

Promoted

Principal Data Engineer

PriceEasy AIBengaluru, Karnataka, India

Join one of the most respected and cutting-edge companies in the pricing and analytics industry! PriceEasy, a VC-backed leader, empowers convenience retailers with actionable insights through advan...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

thinkbridgehosur, tamil nadu, in

We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Digitalzonehosur, tamil nadu, in

As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics. Develop and maintain scalable ETL / ELT pipelines using Py...Show moreLast updated: 7 days ago

Promoted

Data Engineer

Insight Globalhosur, tamil nadu, in

GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts. Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago

Promoted

Lead Data Engineer

Saras Analyticshosur, tamil nadu, in

We are an ecommerce focused end to end data analytics firm assisting enterprises & brands in data driven decision making to maximize business value. Our suite of work spans extraction, transformatio...Show moreLast updated: 2 days ago

Promoted

Senior Data Engineer

SGS & Cohosur, tamil nadu, in

Position Title : Senior Data Engineer.Experience Required : 8 to 12 Years.We are looking for a highly skilled and experienced Data Engineer with strong expertise in. The ideal candidate will play a ke...Show moreLast updated: 7 days ago

Promoted

Principal Data Engineer

K2 Partnering SolutionsBengaluru, Karnataka, India

Job Title : Principal Data / App Engineer, Celonis.Work Location : Bengaluru, Karnataka 560058.Job Type : Full-time / Permanent with the Client directly. Our Client’s Center for Process Excellence team is ...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Tata Consultancy Serviceshosur, tamil nadu, in

Required Technical Skill Set -.Create and maintain optimal data pipeline architecture,.Assemble large, complex data sets that meet functional / non-functional business requirements.Identify, design...Show moreLast updated: 30+ days ago

Promoted

Principal Data Engineer

AutodeskBengaluru, Karnataka, India

We are seeking a highly experienced.Autodesk’s Analytics Data Platform (ADP).This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-o...Show moreLast updated: 18 days ago

Promoted

Principal Data Engineer

MakeMyTripbangalore, karnataka, in

We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences.Nurtured from the seed o...Show moreLast updated: 30+ days ago

Promoted

Lead Data Engineer

Cimpresshosur, tamil nadu, in

Our Team : Enterprise Business Solutions.Vista’s Enterprise Business Solutions (EBS) domain is working to make our company one of the most data-driven organizations to support Finance, Supply Chain,...Show moreLast updated: 7 days ago

Promoted

Data Engineer

IntraEdgehosur, tamil nadu, in

Python, PySpark, AWS services (Glue, Lambda), and Snowflake.The ideal candidate will design, build, and maintain scalable data pipelines, ensure efficient data integration, and enable advanced anal...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Response Informaticshosur, tamil nadu, in

AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture. CICD - Terraform or CloudFo...Show moreLast updated: 30+ days ago