A local model agent played 40 moves of one ARC-AGI-3 game on a studio test machine. It took 230 minutes. It cleared nothing, it never died, and for one of the 40 moves it produced no answer at all. A first time human clears the same level in 22 moves.
That is the per move design, measured end to end, and it is the clearest number we have for why the people winning this benchmark with local models do not build agents this way.
How the local model agent was run
The game was ls20, the toolkit’s first public environment, played in offline mode. The model was gemma-4-31b-qat through LM Studio at 64K context, the one model this stack keeps loaded. Each turn it received the board downsampled to 16 by 16 as hex rows, the four legal actions, and a one line history of its last six moves and whether each one changed the board or ended the game. It was asked for one action name. The cap was 8,192 output tokens, four times what the same board had needed in earlier runs. The loop was the 40 line agent with the random policy replaced by the model call, budgeted at 40 moves or the first cleared level, whichever came first.
Prompt size stayed between 352 and 379 tokens for the whole run, so nothing about the cost below is the prompt.
What 40 moves cost in wall clock and reasoning
Two hundred and thirty minutes for 40 moves, a mean of 345 seconds each. The fastest move took 158 seconds, the median 320, the slowest 736. Reasoning per move ran from 1,774 tokens to 8,188, median 3,602, and the run spent 154,673 tokens of thought in total to press 40 keys.
| Per move | Seconds | Reasoning tokens |
|---|---|---|
| Fastest | 158.3 | 1,774 |
| Median | 320.4 | 3,602 |
| Slowest | 735.6 | 8,188 |
The slowest move is the one that failed. Move 38 hit the 8,192 cap with finish_reason: length and an empty answer, and the harness fell back to ACTION1. Everything the empty reply piece says about caps applies here in miniature: a cap four times the typical need was still not enough for one move in forty, because the spread of thinking on identical inputs is several times, not a few percent.
Against the human baseline the arithmetic is short. Twenty two moves at the median cost is 117 minutes, and the agent had used 40 moves and 230 minutes without reaching the end of level one. Under the scoring, which the methodology page defines as the squared ratio of human to agent actions per completed level, an uncompleted level is zero regardless of effort, so the run’s score is zero, and it would have needed to finish in under 22 moves to score 100 and in under 220 to score above one.
Why the local model agent never died and never progressed
The move sequence explains both. The model pressed ACTION1 for its first eleven turns, then drifted through ACTION2 and ACTION3, and reached ACTION4 only at move 37. Sixteen of the forty moves were ACTION1, twelve ACTION2, nine ACTION3, two ACTION4, and one was the fallback. Random play on the same game, which spreads its presses evenly and resets after every death, died 598 times in 60,000 actions; this agent died zero times in 40, because it spent most of the run pressing one key at the edge of a board that does not punish that.
The history line did not help as much as it should have. It told the model that a move had changed the board or had not, and the model still repeated a move that changed nothing eleven times. Whatever it was reasoning about for 3,600 tokens a turn, it was not the six words of feedback that mattered most.
None of that is a verdict on the model’s intelligence. It is a verdict on the loop: one frame, one instruction, one move, no memory beyond six lines, on a game whose first level a person solves in 22 presses by watching what the pieces do. The design gives the model no way to accumulate what it learns, and it charges seven minutes a turn for the privilege.
What a random agent did with the same game in 15 seconds
The random baseline on ls20 cleared level one at action 40,699 on its first seed, in about 15 seconds of wall clock, after 598 deaths and resets. Its score for the level, 0.000029, is effectively zero, because it took 1,850 times the human’s moves.
Put the two side by side and the comparison is not flattering to either. The random agent completed the level and earned nothing. The model agent earned nothing and did not complete the level, at 900 times the wall clock. The technical report computes the chance of a random win on this level as “exactly 1 in 355” per attempt, and the model, with a whole 31B of reasoning behind each press, did not reach even that lottery’s odds inside its budget, because it never reset and never explored.
An agent that combined the two, the random agent’s willingness to try every key and reset, with the model asked to look at the result once every few dozen moves rather than every move, would cost a fraction of this run and could not do worse.
What this says about per move agents on local hardware
The winning ARC-AGI-3 entries ran models of this size locally, and none of them asked the model for a keystroke. One had it write Python into a live console that then played; another rendered frames and asked for structured output at a rate it could afford. Both moved the per move decision out of the model, and this run is the measurement of why: at five and a half minutes a move, one level of one game is an afternoon, and playing the 25 public games at the median human length of 638 moves each would be about 16,000 moves, two months of continuous inference to attempt once, at human efficiency.
The Mac this ran on steps the game engine at four thousand actions a second. Every one of those actions is free. The only thing that costs anything is the model, and a per move design spends the expensive thing on the cheap thing’s job.
So the number to carry from this run is not 230 minutes. It is 154,673 tokens of reasoning to press 40 keys, and the realisation that the same tokens, spent once on writing a policy the engine could run at four thousand presses a second, would have played the whole game before this agent made its second move.