Observability, Quality & Monitoring Engineer

ROLE AT A GLANCE

Job Title

Observability, Quality & Monitoring Engineer

Department

Technology / AI Enablement

Location

Offshore (Remote)

Employment Type

Full-Time

Experience Required

Minimum 3 years in AI/ML or related observability role

Reports To

AI Platform Lead / VP of Technology

POSITION OVERVIEW

Five Below is seeking a driven and technically skilled Observability, Quality & Monitoring Engineer to join our growing AI Enablement team. In this offshore role, you will be responsible for building and maintaining the systems that ensure our AI platform operates with full transparency, governance, and cost-efficiency. You will serve as the operational backbone of our AI access infrastructure — tracking consumption, enforcing compliance, and providing real-time visibility across all AI services used within the organization.

This is a high-impact role that sits at the intersection of AI engineering, DevOps, and enterprise governance. The ideal candidate brings a strong background in observability platforms and AI tooling, with a keen eye for data quality, system health, and auditability.

KEY RESPONSIBILITIES

AI Access Enablement & Governance

• Oversee the provisioning and lifecycle management of AI access controls, including API keys, model permissions, and user entitlements across AI platforms (e.g., OpenAI, Anthropic, Azure OpenAI, AWS Bedrock).

• Implement and enforce role-based access control (RBAC) policies for AI services in alignment with enterprise security standards.

• Partner with security and identity teams to ensure access governance processes meet internal policy and regulatory requirements.

• Manage onboarding and offboarding workflows for AI tool access, ensuring timely provisioning and deprovisioning.

Observability & Monitoring

• Design, implement, and maintain observability frameworks for AI systems, including logging, tracing, and real-time alerting pipelines.

• Build dashboards and monitoring solutions (e.g., using Datadog, Grafana, Prometheus, or similar tools) to provide end-to-end visibility into AI service health, latency, and error rates.

• Proactively identify anomalies, performance degradations, and reliability risks across AI workloads and escalate appropriately.

• Develop and maintain SLOs/SLAs for AI services and report on performance against defined thresholds.

Consumption Tracking & Cost Management

• Track and report on AI token usage, API call volumes, and resource consumption across teams and projects.

• Build cost attribution models to allocate AI spend by team, product, or use case, enabling data-driven budgeting decisions.

• Identify cost optimization opportunities such as model selection, prompt efficiency improvements, caching strategies, and throttling policies.

• Generate regular consumption reports and present findings to stakeholders and leadership.

Compliance & Auditability

• Maintain comprehensive audit logs for all AI interactions, ensuring data lineage, traceability, and tamper-proof records are in place.

• Ensure AI usage aligns with data privacy regulations (e.g., GDPR, CCPA) and internal data handling policies.

• Support internal and external audits by providing timely access to usage logs, access records, and compliance documentation.

• Develop and maintain policies and runbooks for AI observability, incident response, and compliance workflows.

Quality Engineering

• Establish quality benchmarks for AI model outputs, including accuracy, latency, and consistency metrics.

• Design automated quality checks and regression testing frameworks for AI-powered features and pipelines.

• Collaborate with data science and engineering teams to define and monitor model drift, bias detection, and output quality standards.

• Document quality assurance processes and contribute to continuous improvement initiatives across the AI platform.

REQUIRED QUALIFICATIONS

• Minimum 3 years of professional experience working with AI/ML systems, LLMs, or AI platform infrastructure.

• Demonstrated experience with observability tools such as Datadog, Splunk, Grafana, Prometheus, Open Telemetry, or equivalent platforms.

• Hands-on experience with at least one major AI/LLM provider API (OpenAI, Anthropic, Google Vertex AI, Azure OpenAI, or AWS Bedrock).

• Proficiency in Python or another scripting language for automation, data processing, and tooling development.

• Experience implementing logging, monitoring, and alerting pipelines in cloud environments (AWS, Azure, or GCP).

• Familiarity with identity and access management (IAM) concepts and role-based access control for cloud or AI services.

• Strong understanding of cost tracking and FinOps principles in cloud or SaaS environments.

• Experience maintaining audit trails and supporting compliance requirements (GDPR, SOC 2, or similar frameworks).

• Excellent written and verbal communication skills in English, with the ability to work effectively in an offshore/remote model.

PREFERRED QUALIFICATIONS

• Experience with LLMOps or MLOps platforms (e.g., MLflow, Weights & Biases, LangSmith, or Helicone).

• Familiarity with AI governance frameworks and responsible AI practices.

• Experience building internal developer portals or self-service tooling for AI access management.

• Background in retail technology, e-commerce, or enterprise software environments.

• Certifications in cloud platforms (AWS, Azure, or GCP) or AI/ML specializations.

• Knowledge of vector databases, embedding pipelines, and RAG (Retrieval-Augmented Generation) architectures.

• Experience with infrastructure-as-code tools such as Terraform or Pulumi.

KEY SKILLS & COMPETENCIES

Technical Skills

Professional Skills

AI/LLM API Integration

Observability & Monitoring Platforms

Cloud Infrastructure (AWS/Azure/GCP)

Python / Scripting & Automation

Cost Attribution & FinOps

Audit Logging & Compliance

IAM & Access Control

Quality Engineering & Testing

Analytical & Detail-Oriented Thinking

Strong Written & Verbal Communication

Proactive Problem Solving

Cross-functional Collaboration

Self-directed & Accountable in Remote Settings

Adaptability to Fast-Changing AI Landscape

Skills Required
finop , Monitoring, Api, AI ML, Cloud, Python