Quality Engineer - Big Data Testing

Job Description : Quality Engineer (Data)

JOB SUMMARY :

We are seeking a highly skilled Quality Engineer with 5-10 years of professional experience to ensure the integrity, reliability, and performance of our data pipelines and AI / ML solutions within the SmartFM platform. The ideal candidate will be responsible for defining and implementing comprehensive quality assurance strategies for data ingestion, transformation, storage, and the machine learning models that generate insights from alarms and notifications received from various building devices. This role is crucial in delivering high-quality, trustworthy data and intelligent recommendations to optimize facility operations.

ROLES AND RESPONSIBILITIES :

• Develop and implement end-to-end quality assurance strategies and test plans for data pipelines, data transformations, and machine learning models within the SmartFM platform.
• Design, develop, and execute test cases for data ingestion processes, ensuring data completeness, consistency, and accuracy from various sources, especially those flowing through IBM StreamSets and Kafka.
• Perform rigorous data validation and quality checks on data stored in MongoDB, including schema validation, data integrity checks, and performance testing of data retrieval.
• Collaborate closely with Data Engineers to ensure the robustness and scalability of data pipelines and to identify and resolve data quality issues at their source.
• Work with Data Scientists to validate the performance, accuracy, fairness, and robustness of Machine Learning, Deep Learning, Agentic Workflows, and LLM-based models. This includes testing model predictions, evaluating metrics, and identifying potential biases.
• Implement automated testing frameworks for data quality, pipeline validation, and model performance monitoring.
• Monitor production data pipelines and deployed models for data drift, concept drift, and performance degradation, setting up appropriate alerts and reporting mechanisms.
• Participate in code reviews for data engineering and data science components, ensuring adherence to quality standards and best practices.
• Document testing procedures, test results, and data quality metrics, providing clear and actionable insights to cross-functional teams.
• Stay updated with the latest trends and tools in data quality assurance, big data testing, and MLOps, advocating for continuous improvement in our quality processes.

REQUIRED TECHNICAL SKILLS AND EXPERIENCE :

ADDITIONAL QUALIFICATIONS

(ref : hirist.tech)