What a benchmark score can tell you

Historical report — Stanford’s April 13, 2026 AI Index reported roughly 66% task success on OSWorld, alongside a best-model score of 50.1% for reading analog clocks. Performance depended on the task being measured. ^[1]

OSWorld’s 2024 research paper described 369 tasks spanning desktop and web applications, file operations and workflows across programs. Tasks had specified initial states and execution-based evaluation scripts. That structure made results reproducible within a defined environment; it was not a sample of everything a person might ask an assistant to do. ^[2]

METR’s March 2025 research proposed a different measure: the human task duration at which an agent reached 50% success. It focused on software tasks. The duration referred to human professionals’ completion time, rather than how long a deployed model could run reliably without supervision. ^[3]

By May 19, 2026, METR’s follow-up reported that only five tasks in its TH1.1 suite were estimated to take humans longer than 16 hours. That scarcity limited precision for long time horizons. More capable systems were making parts of the evaluation less informative, rather than eliminating uncertainty about performance. ^[4]

The Index also found widespread capability-score reporting but uneven reporting of responsible-AI benchmarks. A capability result alone did not supply that missing evaluation. ^[1]

About this item: Retrospective report. No first-hand reporting or independent testing is claimed. Corrections policy.