Data Engineer - Validation & Quality

the Role :

We are looking for a Data Engineer Validation & Quality to ensure that every dataset inside Perceive Now is verifiable, accurate, and audit-traceable. You will architect quantitative validation frameworks, build contradiction and anomaly detection systems, and integrate automated evidence scoring mechanisms into our 25-layer data reasoning pipeline.

Responsibilities :

• Design and implement validation frameworks using Python (Pandas, NumPy, Polars) for data quality enforcement, schema validation, and field-level consistency checks.
• Build contradiction-detection and reconciliation pipelines leveraging rule-based systems, cosine similarity, and statistical control models.
• Develop automated confidence scoring models for each record or Evidence Bundle, integrating factors like source reliability, freshness, and duplication metrics.
• Orchestrate validation jobs through Temporal / Airflow / Prefect, ensuring deterministic execution and full observability.
• Automate checksum verification, schema drift detection, and data sampling across hundreds of data sources.
• Create and maintain lineage graphs and quality dashboards in PostgreSQL, OpenSearch, and Grafana for continuous visibility.
• Collaborate with Kernel and Governance pods to embed validation metadata and scoring outputs directly into evidence objects.
• Ensure compliance with enterprise-grade data governance and security frameworks (SOC 2, GDPR, ISO 27001).

Required Qualifications :

Preferred Skills :

Key Performance Metrics :

(ref : hirist.tech)