Principal AI Engineer - Large Language Models

Job Description :

What you'll do :

- Lead the AI team: set the technical vision and direction, establish how we build AI, and grow and mentor a team of engineers.

- Architect and build large-scale agentic systems that automate extremely complex, multi-step healthcare processes along with the foundational frameworks, shared tooling, and evaluation infrastructure that accelerate every product the team builds.

- Stay deeply hands-on as the team's top technical expert, personally solving the hardest problems and making the key architecture and approach calls (agent design, RAG, prompting, fine-tuning).

- Set the bar for reliability and safety: evaluation (including LLM-as-a-judge), guardrails, PHI-safe handling, and clinical safety non-negotiables before anything ships.

- Own the AI roadmap and priorities, and translate company and clinical strategy into what the team builds.

- Partner with the deployment/infrastructure engineer to ensure foundational systems deploy, scale, and run reliably in production.

- Work directly with product, clinical, and executive stakeholders as the technical voice for AI.

- Raise the engineering bar across the team and set the culture for how AI is built here.

What we're looking for :

- 8+ years building AI/ML systems, with deep, current expertise in modern LLM and agentic systems.

- Experience leading engineers or teams setting technical direction, mentoring, and growing people while remaining hands-on.

- A track record of architecting and shipping large-scale agentic systems for complex, real-world workflows plus the foundational frameworks other engineers build on - not just individual applications or demos.

- Recognized technical depth: you're the person a team turns to for the hardest AI problems.

- Strong software and systems engineering fundamentals in Python + your stack; you reason fluently about reliability, scale, and failure modes.

- Product sense and the ability to prioritize the AI bets worth making - and kill the ones that aren't.

- Excellent communication; you can be the technical voice for AI with clinical and executive stakeholders.

AI & LLM engineering depth (core to this role) :

This is the heart of the role. You should have deep, hands-on expertise designing and building large-scale, reliable agentic systems for complex, multi-step workflows. We don't expect every tool or technique below - but we do expect genuine, demonstrable depth in agent architecture and evaluation, and fluency across the rest of the modern stack.

Agent architecture & orchestration - expert level :

- Designing single- and multi-agent architectures for complex, long-running, multi-step processes: orchestration patterns such as planner-executor, supervisor/worker, hierarchical agents, ReAct, reflection, and plan-and-solve.

- Deep, hands-on experience with modern agent frameworks - for example LangGraph, LlamaIndex, CrewAI, AutoGen, the OpenAI Agents SDK, DSPy, Pydantic AI, or Semantic Kernel - and the judgment to weigh their tradeoffs and know when to build your own.

- Agent state and memory: short- and long-term memory, checkpointing, and managing state reliably across long-running processes.

- Tool and function calling, tool design, and emerging standards such as the Model Context Protocol (MCP).

- Context engineering: managing context windows, in-agent retrieval, and decomposing hard problems into reliable steps.

- Engineering for reliability at scale: planning, error recovery and self-correction, human-in-the-loop checkpoints, guardrails, retries, timeouts, and graceful fallback.

Evaluation & reliability - expert level :

- LLM-as-a-judge / model-graded evaluation, plus agent-specific evaluation: scoring trajectories and tool use, not just final outputs.

- Building golden datasets, offline and online evals, regression testing, and red-teaming - owned as core infrastructure, not an afterthought.

- Evaluation and observability tooling - for example LangSmith, Langfuse, Braintrust, Ragas, DeepEval, or Arize Phoenix.

Retrieval & knowledge - deep:

- Advanced RAG: hybrid search, reranking, agentic and graph-based retrieval, and thoughtful chunking and embedding strategies.

- Vector databases (e.g., pgvector, Pinecone, Weaviate, Qdrant, Milvus) and the tradeoffs between them.

Models & adaptation - deep:

- Working fluently across frontier APIs (Anthropic, OpenAI, Google) and open models (Llama, Mistral, Qwen), with clear judgment on model selection.

- Model adaptation when it's justified: fine-tuning and LoRA/PEFT, supervised fine-tuning, and an informed view on preference tuning (DPO/RLHF) and distillation - including when not to fine-tune.

- Structured outputs, function schemas, constrained decoding, and prompt optimization (e.g., DSPy).

Scale, cost & safety - deep:

- Designing agentic systems that hold up at real production scale and volume, reasoning about latency, cost, caching, routing, batching, and streaming. (Serving and infrastructure are owned jointly with the deployment engineer, but you understand the tradeoffs deeply.)

- Hallucination mitigation and grounding, guardrails, PII/PHI redaction, and content safety - essential for clinical A solid working understanding of how modern LLMs actually work - tokenization, attention, context, and sampling/decoding - enough to reason from first principles rather than only calling APIs.

(ref:hirist.tech)