Saturday, 21 November 2026, story 4 of 7
METR: AI gains continue on verifiable tasks, lag on judgment work
The nonprofit's updated time horizon report adds longer, open-ended tasks. Models fall short of humans on designing evaluations and scoring philosophical arguments.

What happened
METR, the AI evaluation nonprofit, has updated its time horizon plot, MIT Technology Review reports. The original plot rated models by how long the tasks they could complete took humans, and that length doubled every four months for frontier models. The update adds longer and more open-ended tasks. The review says progress on verifiable tasks does not seem to be slowing.
On a new task asking models to design AI evaluations for METR's research, METR employees judged none of the results useful. Models also fell short of humans at scoring philosophical arguments. Some newer models, such as Opus 5.5, reach "borderline above the bar" on evaluation design, where a year ago every model rated below the bar.
What it means
For builders, the report supports a split that many already practise: give agents work with an automatic pass/fail check, such as code with tests, and keep people on tasks where success is a matter of judgment. Designing evaluations is itself one of those tasks, so handing it to an agent unreviewed looks premature on this evidence.
The judgment results are thin. Most completion times are estimates, one person rated most of the philosophical arguments, and METR includes few such tasks. Researchers quoted say the improvement is real but hard to extrapolate. The move from "clearly below" to borderline is a signal to re-test, not a reason to delegate. The trend to watch is whether METR can add more tasks and raters.
What to do: Keep agents on work you can check automatically, keep human review on judgment calls, and re-test newer models on your own open-ended tasks before trusting them.
Source
- The most misunderstood graph in AI just got a major update, MIT Technology Review
Filed under Research. Published .
"What it means" is analysis, not reporting. Found an error? Write to daily@vibitely.com.