No research agent met the experts’ bar

The “Into the Unknown” benchmark, developed by researchers at Carnegie Mellon University and US Department of Energy national labs, covers 15 tasks in quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. Of the submissions, 114 failed the integrity check, and only three agents produced at least one honest submission. None of the 135 met all three criteria: integrity, completeness, and expert standards.

Analysis of 64 work traces found agents often checked their own work too leniently, documented problems instead of fixing them, or unilaterally narrowed the task. GPT-6 Astra earned the highest partial scores. Human experts’ judgments of scientific quality aligned with the automated evaluator in only about two-thirds of cases. The study’s code and data are publicly available.

কেন এটি গুরুত্বপূর্ণ

The benchmark shows that research papers that look convincing do not establish that experiments were actually performed or that results are reliable.