State Farm uses generative AI agents to help its chat center representatives answer a high volume of customer questions, such as billing queries. Pre-deployment testing, including LLM-as-a-judge and BERTScore regression tests in CI/CD, could not cover how thousands of real users behave, expert review reached less than 1% of messages, and thumbs-up/down feedback was sparse and hard to act on.
The engineering team built continuous production monitoring with an LLM as judge. On a regular cadence, a scheduled job pulls conversation logs from cloud storage, draws a statistically significant sample (for example 95% confidence, plus or minus 2%) and queues it. A serverless function running the DeepEval framework sends each conversation to Claude Sonnet 4.5 on Amazon Bedrock, which returns a score and written reasoning per metric; results are stored for trend analysis and can also be run on demand for incidents or change validation. Because production has no ground truth, the team relies on reference-free metrics such as faithfulness, contextual relevancy and topic adherence. A run of 1,500 conversations with four metrics costs roughly US$90 to US$150.
Standard metrics missed a key failure. The billing agent scored 0.95 or higher on faithfulness, yet it was telling representatives that accounts were in "good standing" or that customers had "been paying on time", inferences not stated in the retrieved data. Using the GEval pattern, the team wrote a stricter faithfulness metric that treats any claim not explicitly supported by context as unsupported. It surfaced the pattern, and a small prompt change tightening the agent's rules on inference lifted the custom faithfulness score by 8% on the next run. Further custom metrics check whether a question is answerable from the retrieved context and whether responses follow brand tone. A meta-analysis step uses a separate LLM call to read scores and reasoning across a batch and produce structured findings on systemic issues.
To test whether the judge could be trusted, subject matter experts (up to three per conversation) scored 197 real production conversations with the same rubric. When human scores were averaged, 78% of judge scores differed by less than 3 points on a 0 to 10 scale, with a mean absolute difference of 2.14 and no systematic bias (mean directional difference 0.10). The judge caught subtle data misrepresentations humans missed, while humans caught domain distinctions and retrieval gaps the judge could not, so State Farm uses both.