ML Data Pipeline Architect

Lead Data Engineer

Location : Banglore / Pune (Hybrid)

Mode : Hybrid

Shift Timing : 2 PM to 11 PM

Experience : 7-12 years. 3+ years in cloud data platforms and Databricks.

Purpose :

We are seeking a hands-on Technical Lead to drive the ingestion of high-volume mainframe RPC data into Databricks , enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.

Key Responsibilities

• Mainframe Data Ingestion :
• Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.
• ML-Ready Data Engineering :
• Transform and structure data for time-series modelling and anomaly detection across thousands of models.
• ML Workflow Integration :
• Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types;

enable automated retraining and scoring.

implement auto-scaling and cost-aware job scheduling.

Required Skills

Preferred Skills