Data Engineering Manager

iMerit is a leading AI data solutions company that transforms unstructured data into structured intelligence for advanced machine learning and analytics. Our customers span autonomous mobility, medical AI, agriculture, and more delivering high-quality data services that power next-generation AI systems.

About the Role

We are looking for a seasoned Engineering Lead to architect, scale, and continuously evolve our analytics and observability platform—a system deeply integrated with annotation tools and ML pipelines. This platform powers real-time visibility, operational insights, and automation across large-scale data operations.

In this role, you will not only lead and mentor a team but also set the technical vision for high-throughput streaming systems and modern data lake / warehouse architectures. You will bring proven expertise in high velocity, high volume data engineering, driving innovation in how we process, curate, and surface data to support mission-critical AI workflows

Key Responsibilities

Lead & Inspire : Build and mentor a high-performing data engineering team, fostering innovation, accountability, and technical excellence

Architect at Scale : Design and implement high-volume batch and real-time data pipelines across structured and unstructured sources

Build and maintain real-time data lakes with streaming ingestion, ensuring data quality, lineage, and availability.

Curate, transform, and optimize datasets into high-performance data warehouses (e.g., Redshift, Snowflake) for downstream analytics

Deep Streaming Expertise : Drive adoption and optimization of Kafka for messaging, event streaming, and system integration, ensuring high throughput and low latency

Advanced Processing : Leverage PySpark for distributed data processing and complex transformations, delivering scalable ETL / ELT pipelines

Orchestration & Automation : Utilize AWS Glue and related cloud services to orchestrate data workflows, automate schema management, and scale pipelines seamlessly

Continuous Improvement : Oversee platform upgrades, schema evolution, and performance tuning, ensuring the platform meets growing data and user demands

Observability & Insights : Implement metrics, dashboards, and alerting for key KPIs (annotation throughput, quality, latency), ensuring operational excellence

Cross-Functional Collaboration : Work closely with product, platform, and customer teams to define event models, data contracts, and integration strategies

Innovation and R&D : Research emerging technologies in data streaming, lakehouse architectures, and observability, bringing forward new approaches and prototypes

Minimum Qualifications

10+ years of experience in data engineering or backend engineering, with at least 2–3 years in a leadership or team-lead role

Proven track record in building and operating data pipelines at scale—including both batch ETL / ELT and real-time streaming

Expert-level experience with Kafka for high-throughput data ingestion, streaming transformations, and integrations

Strong hands-on experience with PySpark for distributed data processing and advanced transformations

In-depth knowledge of AWS Glue(or similar) for orchestrating workflows, managing metadata, and automating ETL pipelines

Demonstrated success in upgrading and maintaining real-time data lakes, curating and transforming datasets into performant data warehouses

Familiarity with lakehouse and warehouse patterns (e.g., Delta Lake, Redshift, Snowflake) and schema versioning

Experience with cloud-native data services (S3, Kinesis, Lambda, RDS) and infrastructure-as-code deployments

Preferred Qualifications

Experience with Databricks and Snowflake solutions, including developing on lakehouse architectures and optimizing warehouse performance

Exposure to annotation platforms, ML workflows, or model validation pipelines

Experience with observability tools (Prometheus, Grafana, OpenTelemetry)

Knowledge of data governance, RBAC, and compliance in large-scale analytics environments

Comfort working in Agile, distributed teams with Git, JIRA, and Slack.

Why Join Us?

At iMerit, you will lead a team at the cutting edge of AI data infrastructure—building and evolving platforms that are explainable, auditable, and scalable. You will play a key role in upgrading and maintaining our streaming data lake and transforming it into analytics-ready warehouses, directly shaping how AI systems are built and trusted at scale.