In a recent Princeton-led study, Kirgis et al. gave frontier AI agents the central research questions of two unpublished NeurIPS 2026 papers, along with six days and thousands of dollars’ worth of computing resources. The agents completed all the required coding and experimental engineering without human assistance, yet made no substantial progress on the underlying research problems. When the authors of the original studies evaluated the resulting manuscripts, they awarded them just 2/6 and 1/6 unequivocal rejections. https://arxiv.org/abs/2607.27191
Their failure was not primarily technical. The agents could write code, run experiments and assemble superficially credible manuscripts; what they lacked was scientific judgement. They repeatedly misjudged the standard required for publishable research, responded unimaginatively to weaknesses in their experimental designs, failed to reconsider or abandon unproductive directions, managed resources poorly and gradually lost sight of their original instructions. In other words, they could operate the machinery of research but could not reliably determine which questions were worth pursuing, which evidence mattered or which conclusions were scientifically justified.
A complementary study suggests that these limitations extend far beyond two failed papers. The authors analysed more than 25,000 agent runs across eight scientific domains, finding that available evidence was ignored in 68% of reasoning traces and that refutation prompted agents to revise their beliefs in only 26%. These patterns persisted even when agents were shown nearly complete examples of successful reasoning. Curiously, although this study appeared more than three months earlier and offers a compelling explanation for many of their findings, Kirgis et al. did not cite it, a conspicuous and rather puzzling omission.
Taken together, these findings cast serious doubt on predictions that AI will soon automate science and unleash explosive recursive self-improvement. They do not by themselves settle what we should build instead, but they sharpen the case for a different direction. As I argue in the post Civilizational Futures: Five Normative Imperatives for a Science-Based Civilization, AI should serve as intellectual scaffolding, it should enable humanity to build higher than it could alone while ultimately leaving behind stronger, more independent minds rather than deeper technological dependence. Writing in Nature three days after this post was first published, Nobel laureate Daron Acemoglu makes a parallel case, urging us to abandon the obsession with artificial general intelligence and instead build “pro-worker” AI that amplifies human expertise rather than eliminating the people who possess it. In science, this means using AI to expand researchers’ capacity to formulate hypotheses, interrogate evidence and challenge assumptions not pursuing the premature fantasy of autonomous machine scientists. Until agents can reason seriously about evidence, refutation and uncertainty, AI will remain far better at convincingly imitating the visible production of research than at genuinely reproducing the intellectual process through which reliable knowledge is created.
