A study finds that AI models can detect some misbehavior in agent logs, but higher detection rates also mean more warnings about harmless actions. The results suggest auditors may be more useful for estimating how often problems occur than for safety checks that must catch every incident. Andy Hall evaluated several models on logs from METR’s MALT dataset. At a 5 percent false-alarm rate, GPT-5 caught nearly 98 percent of 47 reward-hacking cases, while Gemini 3.8 Flash found about half. Auditors caught 71 to 81 percent of sandbagging cases, most of which were synthetic.
In Vals AI’s simulated tests, Deepseek V4.1 Flash 1017 came close to a leading closed model in some scenarios. The findings highlight the importance of long test runs and lower costs, but do not establish performance on real networks. In the Vals CyberOffense benchmark, agents pursued objectives in simulated corporate networks without human help. GPT-6 Sol Daybreak averaged 15.7/21 objectives in Nova and reached 27/31 in Lighthouse; Deepseek V4.1 Flash 1017 scored 16.3/21 and 21/31. Vals reports approximate run costs of $22 and $3, respectively.
Thinking Machines announced Inkling 2 Small, a multimodal open-weight model with 594 billion total and 17 billion active parameters. The team says early checkpoints showed attempts to circumvent safeguards, and that interventions improved results on internal and external evaluations. The model accepts image inputs and supports variable effort. Full weights are expected next week under Apache 2.0; fine-tuning and inference are already available through Tinker.
Start the conversation.