Why METR changed its coding-productivity experiment

On February 24, 2026, METR announced changes to its coding-productivity experiment. The problem was selection: developers increasingly avoided participation, or particular tasks, rather than risk having to work without AI. ^[1]

The earlier experiment, published in July 2025, had tested 16 experienced contributors on 246 real issues from repositories they knew well. Issues were randomly assigned to permit or prohibit AI. METR measured a 19% increase in completion time when AI was allowed, even though participants believed it had saved them time. That was a result for a particular setting and generation of tools, rather than a general verdict on coding assistants. ^[2]

In the follow-up, tasks were again randomized after developers submitted them. But 30% to 50% of surveyed developers reported withholding some tasks to avoid an assignment without AI. METR said this could exclude work with particularly high expected gains; lower participation pay could also affect selection. ^[1]

METR thought developers probably gained more speed than before, but called its data very weak evidence of how much. Concurrent agents complicated timekeeping, too: developers could work on something else while an agent ran. ^[1]

The original study’s completion standard included a human developer's judgment that code would pass review, with style, tests and documentation requirements. Automated benchmarks instead used algorithmic scoring. METR cautioned that these methods address different questions: successful benchmark performance does not translate directly into measured time saved in familiar repositories. Its original result also did not establish that other developers could not benefit. ^[2]

Later research pursued another route. In May 2026, METR reported a survey of 349 technical workers conducted in February through April. It asked separately about time savings and the value of work produced. Participants reported median value multipliers between 1.4 and two, depending on question wording, versus a threefold speed multiplier. Those were self-reports, rather than measured productivity gains or a repaired version of February's experiment. ^[3]

The survey illustrates the practical measurement problem for teams weighing AI adoption: faster work can change which tasks people choose, without adding proportionate value. METR also warned that its convenience sample and low email response rate could create selection bias. Internally consistent answers could not establish whether perceptions matched reality. The follow-through broadened the evidence, but left the magnitude of actual productivity gains unresolved. ^[3]

About this item: Retrospective report. No first-hand reporting or independent testing is claimed. Corrections policy.