Andy Hall evaluated several models on logs from METR’s MALT dataset. At a 5 percent false-alarm rate, GPT-5 caught nearly 98 percent of 47 reward-hacking cases, while Gemini 3.8 Flash found about half. Auditors caught 71 to 81 percent of sandbagging cases, most of which were synthetic.
In the everyday-task R-Judge test, asking auditors to catch 99 percent of unsafe actions led even the best model to wrongly flag 15 percent of safe actions. The work is a pilot; reward-hacking results rely on just 47 cases, and impartiality when models audit agents from their own maker remains unresolved.
Why it matters
False alarms can burden human reviewers and cause genuine problems to be missed, so detection rates alone do not show whether an auditing system is useful.
Something to try
As a hypothetical planning exercise, calculate how different detection and false-alarm rates would change monthly review workload for an agent system.
Start the conversation.