Each task family is a ladder of seven difficulty levels. We run every model as a working agent, twice per task, sealed off from the internet, and mark the level where it stops solving half the tasks.
Filled dot: the level where it still solves half the tasks every time. Ring: the level where it solves half of them at least once. The thin line is the uncertainty. Faded marks and grey numbers are not yet settled with two tries.
Share of tasks solved at each level. A dashed line is the same model at its highest reasoning setting. Rule finding drops from all solved to almost none in a single step. Load planning fades more slowly.
A model that solves a task once and fails it the next time is a different thing from one that cannot solve it. All 56 tasks, by what happened across the two tries.
Up is better, left is cheaper. Bars show the uncertainty in the score. Tokens are counts and are not equal in price across providers.
Pick a battery and up to four configurations, then choose what to look at. Every bar is drawn to the largest value among the ones you picked.
Solved tries out of valid tries at each level. Darker is more solved.