The Einstein test exposes why proving AI discovery is so hard
Could an AI trained only on knowledge available before a scientific breakthrough rediscover the breakthrough independently? Nature examines that deceptively simple test through historical language models built with cutoff dates before relativity, quantum mechanics, Turing machines, and other landmark ideas. The early results are humbling. A model trained on pre-1900 material showed occasional phrases that resembled later insights after receiving strong hints, but mostly failed and often produced plausible language without a reliable physical model. Other researchers attempting a pre-1930 system discovered that the training corpus leaked later facts: the supposedly historical model could answer questions about Franklin D. Roosevelt's administration. A University of Zurich family of four-billion-parameter models uses cutoffs at 1913, 1929, 1933, 1939, and 1946, but limited historical data and compute constrain what those systems can demonstrate. The test reveals two separate problems. First, dated archives are messy, incomplete, and contaminated by metadata and digitization. Second, a generative model can produce many theories, some suggestive and many wrong, while science still needs a process to rank them and connect them to evidence. Mathematics offers formal verification; empirical science requires experiments, instruments, causal reasoning, and judgment about which hypothesis deserves scarce attention. Historical models remain valuable because they can expose hindsight leakage and benchmark scientific novelty. But a striking rediscovery claim should not count unless the dataset, cutoff, prompts, researcher hints, candidate failures, and evaluation rule are independently reconstructable.
