METR: We are Changing our Developer Productivity Experiment Design

Описание: The research organization METR recently announced significant methodological changes to their evaluations of artificial intelligence-driven developer productivity after recognizing that their latest experimental design yielded unreliable data. Following an initial early 2025 study which suggested that AI tools actually slowed developers down by 19 percent, researchers launched a subsequent trial in August 2025 with a larger participant pool to track these effects over time. However, the escalating capability and widespread adoption of agentic AI tools introduced profound selection bias into the new study, as developers increasingly refused to participate or selectively withheld complex tasks because they were unwilling to work without AI assistance. This severe selection bias, compounded by a reduction in participant compensation from 150 to 50 dollars per hour, caused researchers to conclude that their preliminary data showing minor productivity improvements drastically underestimates the true speedup provided by modern AI. Furthermore, confounding variables such as altered task quality, task abandonment in control groups, and the inherent difficulty of logging time while autonomous agents operate concurrently made the central estimates exceptionally difficult to interpret. Consequently, METR is pivoting away from its original self-selected, task-level randomization approach to explore alternative evaluation frameworks, including short intensive experiments, observational data analysis, and fixed-task assignments, to more accurately measure the evolving impact of advanced AI systems.

https://metr.org/blog/2026-02-24-upli...