
The research agent had to predict before it saw the answers
The interesting part of this experiment came before the researchers opened the test data. In a blinded evaluation, a biomedical research agent produced a six-protein score and its prediction rule was locked. Only then was it evaluated against survival outcomes in a separate group of 176 people. There were 17 deaths in that group. The reported concordance index was 0.826: the score tended to order comparable people by risk in the right direction. It does not mean the system correctly diagnosed 82.6 percent of patients. OpenScientist takes questions specified by scientists and works through analyses. The Nature Medicine paper describes five biomedical cases, with experts reviewing the work. The protein example is especially useful because holding back the answers changes the incentives. Once researchers have seen the outcomes, it is easy to adjust a plausible explanation until it fits. A prediction written down ahead of time has to take its chances. There is a payoff here even if no clinician ever uses this particular score. A research group could ask an agent to propose an analysis, preserve that version, and let another group test it. Seventeen deaths are a thin basis for deciding how to treat an individual. An external cohort with different patients, a calibration check, and a comparison against current practice would tell us much more about whether anyone should act on the ranking.