Why it matters
benchmark automation itself becomes a governance problem when models are assessed for clinical use.
Go to the source
Read the evidence behind this analysis. External links open in a new tab.
NatureCognition & learningTechnical failuresLaw & informationSocial good & healthGlobalResearch
Shi et al., “Physicians and artificial intelligence diverge in evaluating LLMs on real clinical cases”
This multicenter study involved more than 400 physicians across seven specialties and compared human physician evaluation of LLM outputs with AI-agent evaluation configured to mirror physician assessment. AI evaluators were efficient and directionally aligned with physicians, but did not fully capture human clinical judgment and should not replace physician-centered evaluation.
The impact in one sentence
This multicenter study involved more than 400 physicians across seven specialties and compared human physician evaluation of LLM outputs with AI-agent evaluation configured to mirror physician assessment.
