VideoZeroBench tests video AI’s ability to show its evidence
VideoZeroBench tested models on 500 questions from 138 long videos, checking whether they could identify not only answers but also where the evidence appears in time and space. According to the paper, the top standard answer score was 17%; no model exceeded 1% when evaluation also required correct temporal and spatial grounding.
AlphaSignalNov 7, 2026
1 min read
The benchmark uses a five-level protocol that separates answer accuracy from temporal and spatial evidence requirements. It includes temporal evidence annotations for 442 questions and spatial evidence for 372; videos average 667 seconds.
The authors report that manually zooming into evidence regions raised Gemini-3-Pro’s score above 40%. They say this points to evidence retrieval as a possible bottleneck. Direct comparison with earlier benchmarks is limited by differences in task format and difficulty.
Why it matters
Scoring answers alone does not show whether a model relied on relevant video evidence. This benchmark evaluates answers and their visual support separately.
GUIDES
Something to try
Hypothetically evaluating a video analysis system, track answer accuracy and the cited evidence interval as separate measures.
PUSULAM / UNDERSTAND
A good question changes everything.
Let’s unpack this article together. Ask what it means, why it matters, or how to use the idea.
A personal overviewFor your work and interests
What does this mean for you?
An explanation shaped around your work and interests.
Tell us a little about yourself to get an explanation that fits.
PUSULAM / COMMUNITY
The conversation
What did this change for you? Share a useful example or ask other readers a question.
AutoTrust Labs released JEV-27B v1.1, a 27-billion-parameter model that returns option probabilities rather than free-form text. The company reports a score of 67.31 across 37 benchmarks; the model supports multiple-choice and yes/no questions only. The model selects among supplied options and returns a probability distribution. It stands out on tool selection and intent classification, but is weaker on open-ended reasoning and some other tests.
A study finds that AI models can detect some misbehavior in agent logs, but higher detection rates also mean more warnings about harmless actions. The results suggest auditors may be more useful for estimating how often problems occur than for safety checks that must catch every incident. Andy Hall evaluated several models on logs from METR’s MALT dataset. At a 5 percent false-alarm rate, GPT-5 caught nearly 98 percent of 47 reward-hacking cases, while Gemini 3.8 Flash found about half. Auditors caught 71 to 81 percent of sandbagging cases, most of which were synthetic.
In Vals AI’s simulated tests, Deepseek V4.1 Flash 1017 came close to a leading closed model in some scenarios. The findings highlight the importance of long test runs and lower costs, but do not establish performance on real networks. In the Vals CyberOffense benchmark, agents pursued objectives in simulated corporate networks without human help. GPT-6 Sol Daybreak averaged 15.7/21 objectives in Nova and reached 27/31 in Lighthouse; Deepseek V4.1 Flash 1017 scored 16.3/21 and 21/31. Vals reports approximate run costs of $22 and $3, respectively.
Start the conversation.