Lessons from deploying the ChatEHR system at Stanford Medicine

In deploying the ChatEHR large language model at Stanford Medicine, the authors found benchmark-based evaluations insufficient for monitoring clinician-driven interactions. The piece argues new methods for monitoring performance are needed in large medical center deployments.