AI QA Automation Engineer

Title : AI QA Automation Engineer

Location : Remote

Job Summary :

We are seeking an AI Quality Engineer with a strong automation skillset to ensure the robustness,

performance, and reliability of our AI systems and services. The ideal candidate is tech-savvy,

proactive, and passionate about quality at every step—from initial design through deployment and

ongoing monitoring. You will play a key role in building and maintaining a highly automated testing

infrastructure to support fast, reliable model and pipeline delivery as the company scales.

Key Responsibilities

Testing Expertise :

• Conduct comprehensive testing across all layers, including server load, integration points, and

output quality.

before the start of feature development.

Automation-First Approach :

performance, and end-to-end testing.

and repeatability.

DevOps & CI / CD Integration :

data pipelines) are tightly integrated with modern CI / CD workflows.

Automated Testing Frameworks :

to-end browser testing), Postman (API testing), and Langfuse (LLM output tracking / testing).

services.

Manual & LLM Testing :

Model (LLM) output quality.

requirements.

Monitoring, Observability & Post-Deploy Quality :

 Configure, deploy, and interpret dashboards from monitoring tools like Prometheus,

Grafana, and CloudWatch.

 Track model health, pipeline performance, error rates, and system anomalies after

deployment.

 Proactively investigate and triage quality issues uncovered in production.

Core Abilities and Technical Skills :

model workflows.

as those from LLMs.

automation-driven quality throughout the organization.

Mindset :

production.

industry best practices.

quality models and features at scale.

Show more

Show less

Skills Required

Cloudwatch, Monitoring Tools, Prometheus, Grafana, Postman, Pytest