A study finds that AI models can detect some misbehavior in agent logs, but higher detection rates also mean more warnings about harmless actions. The results suggest auditors may be more useful for estimating how often problems occur than for safety checks that must catch every incident. Andy Hall evaluated several models on logs from METR’s MALT dataset. At a 5 percent false-alarm rate, GPT-5 caught nearly 98 percent of 47 reward-hacking cases, while Gemini 3.8 Flash found about half. Auditors caught 71 to 81 percent of sandbagging cases, most of which were synthetic.
In Vals AI’s simulated tests, Deepseek V4.1 Flash 1017 came close to a leading closed model in some scenarios. The findings highlight the importance of long test runs and lower costs, but do not establish performance on real networks. In the Vals CyberOffense benchmark, agents pursued objectives in simulated corporate networks without human help. GPT-6 Sol Daybreak averaged 15.7/21 objectives in Nova and reached 27/31 in Lighthouse; Deepseek V4.1 Flash 1017 scored 16.3/21 and 21/31. Vals reports approximate run costs of $22 and $3, respectively.
VideoZeroBench tested models on 500 questions from 138 long videos, checking whether they could identify not only answers but also where the evidence appears in time and space. According to the paper, the top standard answer score was 17%; no model exceeded 1% when evaluation also required correct temporal and spatial grounding. The benchmark uses a five-level protocol that separates answer accuracy from temporal and spatial evidence requirements. It includes temporal evidence annotations for 442 questions and spatial evidence for 372; videos average 667 seconds.
Start the conversation.