Sprouts.ai - Data Engineer

Job Summary :

We are looking for an experienced Data Engineer to design, build, and maintain scalable data pipelines and processing systems that power our modern data platform. The ideal candidate will have hands-on expertise in distributed data processing, data lake / lakehouse systems, and workflow orchestration using modern tools such as Spark, PySpark, Apache Iceberg, Apache Hudi, and Apache Airflow.

This role will be central in enabling reliable data ingestion, transformation, and governance-ready pipelines to support analytics, AI, and operational use cases across the organization.

Key Responsibilities :

• Build and maintain scalable ETL / ELT pipelines for batch and streaming data using PySpark and Airflow.
• Develop and optimize data ingestion workflows from multiple structured and unstructured data sources.
• Implement data lake and lakehouse solutions using Spark with Apache Iceberg, Hudi, or Delta Lake.
• Ensure data quality, reliability, and integrity through validation, profiling, and monitoring frameworks and automations using AI.
• Support database migration efforts and cross-system data reconciliation.
• Collaborate with data scientists, analysts, and platform engineers to enable analytics and ML use cases.
• Contribute to data governance practices, including schema evolution, versioned data catalogs (e.g., Nessie), and metadata management.
• Troubleshoot and optimize data jobs for performance and cost efficiency.

Required Skills & Qualifications :

Nice to Have :

Why Join Us :

(ref : hirist.tech)