Numan builds eval-first AI health coach with LangGraph to avoid medical regulation pitfalls

Numan, a UK digital health company, developed Nu, an AI health coach for patients managing long-term conditions. To avoid crossing into medical diagnosis or prescriptions, the team abandoned static single-turn evaluations and scripted multi-turn tests, which failed to capture conversation context and safety boundaries. Instead, they built a custom eval framework using LangGraph and LangSmith, implementing a supervisor model that orchestrates handoffs to domain-specialist sub-agents with categorized rubrics to evaluate safety and coaching quality across full conversation histories.