McLeod Health, a nonprofit nonacademic health system of 7 hospitals and more than 1,200 providers serving 18 counties in North and South Carolina, implemented a structured three-phase ambient AI documentation evaluation that cut clinician time in notes by 28.3% (P<.001, n=23) during the 90-day pilot, with a 35.4% drop in after-hours pajama time that fell just short of significance (P=.054), then reached 81% system-wide adoption with nearly 150,000 notes generated. Established-patient volumes rose 8.5%, associated with a projected revenue gain of US $2,629 per provider per month, and patient satisfaction improved significantly across listening, trust, communication, and treatment information domains (all P<.001).
The pressure to act came from scale constraints common to regional health systems: McLeod operates without the dedicated research infrastructure, centralized innovation programs, or academic funding streams of academic medical centers, while facing thin margins, physician burnout, and recruitment difficulty. Vendor selection in ambient AI is normally undermined by cognitive biases, unvalidated marketing claims, and limited real-world testing, so the health system's chief medical informatics officer led a multidisciplinary team — system and regional CMOs, ambulatory practice leadership, the CIO, revenue integrity and clinical informatics directors, physician champions, and nursing leadership — to build an objective selection process.
The evaluation unfolded in four phases: design began in February 2024 (phase 0); in phases 1-2 (March through June 2024) four leading vendors were tested through live clinical simulations using 15 complex outpatient scripts, with organizational leaders serving as standardized patients, and AI-generated notes scored by physicians, revenue cycle experts, and nonclinical reviewers for accuracy, billing quality, and readability; the top two vendors advanced to demonstrations of Epic workflow integration with physician usability feedback guiding final selection. The chosen vendor then ran a 90-day pilot across 5 ambulatory specialties beginning October 2024, followed by system-wide implementation in January 2025. All statistical comparisons were 2-sided using a 95% CI.
Operationally, coding patterns shifted toward higher-complexity visits with a 3.8% increase in level 4 established patient visits (P=.05), and the satisfaction gains exceeded those of prior system-wide patient satisfaction initiatives. The authors — Bryon Kenneth Frost, MD (McLeod Health Department of Information Technology), with colleagues Victor Eugene Collier, Franklin Sturgill, Jessie Polson, and Jennifer Jones — report that the multiphase process minimized vendor influence and cognitive bias during selection, validated results through real-world clinical testing, and offers a practical framework for other nonacademic health systems to assess, implement, and scale ambient AI solutions while preserving fairness, transparency, and measurable value.