Production AI support agent with earned autonomy
A global career-services group running three consumer brands handled a high volume of repetitive billing, refund, cancellation and account email across 19 locales. Off-the-shelf LLM support bots were rejected as unsafe: a wrong refund promise or a fabricated account fact reaches a real customer.
What we did
- Built a multi-agent system (OpenAI Agents SDK, FastAPI, Postgres) embedded in the client's live support inbox: a triage agent routes to four specialists, each grounded in live account, payment and refund-eligibility data — no canned templates.
- Set the quality bar empirically: statistical analysis of 6,734 real human support conversations produced the scoring rubric and three significant drivers of reply quality, wired directly into prompt compilation.
- Built a four-track evaluation platform (LLM-as-judge G-Eval scorers, deterministic compliance validators, CI gates) plus a three-tier guardrail pipeline for safety, policy and tone.
- Ran six adversarial red-team audits against live production drafts, each closing a specific failure class (fabricated account claims, internal-instruction leakage, over-promised financial actions).
- Designed staged autonomy: write-actions ship feature-flagged and off, earning autonomy per action only after audit evidence. Money never moves automatically.
