AI acceleration’s evidence gap

METR’s November 5 assessment estimated Anthropic’s progress on well-specified tasks was twice its March 2024–October 2025 pace, with substantial uncertainty. Its August–October access included internal benchmarks and staff interviews. ^[1]

METR attributed much acceleration to AI assistance, not fully autonomous research. Anthropic redacted most evidence METR found most informative. Benchmark limitations and non-AI explanations could change the estimate; readers cannot inspect withheld evidence. ^[1]

An October assessment proposal separates three questions: whether capabilities improved faster, what caused that change, and which bottlenecks could limit further growth. Benchmark trends alone do not answer all three. It calls for examining labor, computing, data and algorithms together. ^[3]

METR’s separate transparency policy explains why access terms matter: evaluators should disclose limits, conflicts and redactions, and publish evidence others can analyze. It describes voluntary dependence on developers as a continuing constraint, even when assessors take no company payment. ^[2]

The policy distinguishes gathering evidence from deciding deployment. METR argues that acceptable-risk judgments belong in open discussion, rather than within one organization. ^[2]

About this item: News report. No first-hand reporting or independent testing is claimed. Corrections policy.