
More medical AI agents did not win this comparison
An honest limitation can be more useful than an impressive score. A new Scientific Reports study of emergency-department disposition decisions makes that case unusually clearly. Its retrospective, single-center evaluation covered 469 encounters. An initial high accuracy figure came from repeated refinement on the same cohort, with information that could reveal clinicians' decisions. The authors report substantially weaker results when that information was withheld. A later, pre-run locked protocol still reused the cohort. On discrimination, its multi-agent system's area under the curve was 0.794, below the 0.821–0.823 reported for single-agent comparators. These findings do not establish better patient outcomes, and the published abstract does not justify treating one hospital's records as a prospective deployment trial. That candor deserves appreciation. It helps distinguish a system that performs well with helpful information from one that would be useful when a clinician actually needs a decision. Complexity is not a clinical benefit by itself; extra agents also need to justify their cost, latency and potential disagreements. For hospital buyers, the lesson is to ask for an untouched evaluation set, a credible simple comparator and a test using only information available at the decision point. This paper is not a verdict against medical AI. It is a reason to reward the harder comparison before patients become the next test set.