Search the evidence

Find the signal.

Search titles, impact clusters, countries, organizations and the full text of every analysis.

2 stories found

A researcher places a blank prediction card in a green envelope beside a sealed purple one.
Social good & healthGlobal+1 clusters01

The research agent had to predict before it saw the answers

The interesting part of this experiment came before the researchers opened the test data. In a blinded evaluation, a biomedical research agent produced a six-protein score and its prediction rule was locked. Only then was it evaluated against survival outcomes in a separate group of 176 people. There were 17 deaths in that group. The reported concordance index was 0.826: the score tended to order comparable people by risk in the right direction. It does not mean the system correctly diagnosed 82.6 percent of patients. OpenScientist takes questions specified by scientists and works through analyses. The Nature Medicine paper describes five biomedical cases, with experts reviewing the work. The protein example is especially useful because holding back the answers changes the incentives. Once researchers have seen the outcomes, it is easy to adjust a plausible explanation until it fits. A prediction written down ahead of time has to take its chances. There is a payoff here even if no clinician ever uses this particular score. A research group could ask an agent to propose an analysis, preserve that version, and let another group test it. Seventeen deaths are a thin basis for deciding how to treat an individual. An external cohort with different patients, a calibration check, and a comparison against current practice would tell us much more about whether anyone should act on the ranking.

3 min
Technical failuresGlobal+3 clusters02

Tac, Gardner, and Kuhl, “Generative artificial intelligence creates delicious, sustainable, and nutritious burgers”

Stanford researchers used generative AI trained on 2,216 human-designed burger recipes and 146 ingredients, then sampled one million recipes to optimize taste, environmental impact, and nutrition. In a blinded restaurant sensory evaluation with 101 participants, one mushroom-based formulation had an environmental-impact score more than an order of magnitude lower than the Big Mac benchmark, while a bean-based burger nearly doubled the nutritional score and reduced environmental impact by a factor of six.

2 min