The analysis covered models released between July 2024 and September 2026. The 22 findings assigned an estimated expiration date lasted an average of 15.7 months. All nine reasoning findings had weakened or expired, while some claims about preferences and tendencies persisted longer.
Results also varied widely between models. In the same citation-fabrication test, GPT-6.1 Sol fabricated 0% of references, Claude Sonnet 5.5 1%, and Qwen3.7-Flash 69%. The researchers recommend tying LLM findings to a specific model, version, and date, and publishing prompts and analysis code.
Why it matters
As LLMs change, previously reported capabilities and weaknesses can quickly become outdated, so research claims need version and date context.
Something to try
When reviewing an LLM performance claim, note the model version, test date, and prompts; if possible, rerun the same test on the model you use.
Start the conversation.