Vibitely Daily

Saturday, 21 November 2026, story 4 of 7

METR: AI gains continue on verifiable tasks, lag on judgment work

The nonprofit's updated time horizon report adds longer, open-ended tasks. Models fall short of humans on designing evaluations and scoring philosophical arguments.

Illustration: A staircase of identical blocks doubling in height toward the horizon on one side, beside a foggy field where a few loose stones sit unevenly, with no clear path through.

What happened

METR, the AI evaluation nonprofit, has updated its time horizon plot, MIT Technology Review reports. The original plot rated models by how long the tasks they could complete took humans, and that length doubled every four months for frontier models. The update adds longer and more open-ended tasks. The review says progress on verifiable tasks does not seem to be slowing.

On a new task asking models to design AI evaluations for METR's research, METR employees judged none of the results useful. Models also fell short of humans at scoring philosophical arguments. Some newer models, such as Opus 5.5, reach "borderline above the bar" on evaluation design, where a year ago every model rated below the bar.

What it means

For builders, the report supports a split that many already practise: give agents work with an automatic pass/fail check, such as code with tests, and keep people on tasks where success is a matter of judgment. Designing evaluations is itself one of those tasks, so handing it to an agent unreviewed looks premature on this evidence.

The judgment results are thin. Most completion times are estimates, one person rated most of the philosophical arguments, and METR includes few such tasks. Researchers quoted say the improvement is real but hard to extrapolate. The move from "clearly below" to borderline is a signal to re-test, not a reason to delegate. The trend to watch is whether METR can add more tasks and raters.

What to do: Keep agents on work you can check automatically, keep human review on judgment calls, and re-test newer models on your own open-ended tasks before trusting them.

Source

Filed under Research. Published .

"What it means" is analysis, not reporting. Found an error? Write to daily@vibitely.com.