The best ARC-AGI-2 score in the 2025 Kaggle competition was 24%. The best frontier score on ARC-AGI-3 at its launch six months later was 0.51%. Those two numbers are not on the same scale, do not measure the same thing, and a system’s position on one says almost nothing about its position on the other.
ARC-AGI-2 vs ARC-AGI-3 is not a version bump. Here is what changed, from the two benchmarks’ own reports, and why the scores do not carry across.
What ARC-AGI-2 asks, and what ARC-AGI-3 asks instead
ARC-AGI-2 is a set of puzzles in the format the series has used since 2019. Its paper says it “preserves the input-output pair task format of its predecessor”: a handful of example grids showing a transformation, and one test grid to which the same transformation must be applied. You look at the examples, infer the rule, and produce an output. It is a static task with one right answer.
ARC-AGI-3 is a set of games. Its technical report describes 135 turn based environments with no instructions, where an agent has to press keys, watch what changes, work out what the game wants, and get there. There is no example to study and no single answer to produce. There is a sequence of actions, and the sequence is judged.
The human timings show the gap. The ARC-AGI-3 report notes that a task from ARC-AGI-1 “takes humans about 30 seconds to solve, while a task from ARC-AGI-2 takes humans about 300 seconds”, and that every ARC-AGI-2 task “was human-calibrated with over 400 untrained participants to ensure 100% solvability”. An ARC-AGI-3 environment took its testers a median of 7.4 minutes per attempt, across several levels, with the baseline for each level set by the upper median player rather than the best.
How ARC-AGI-2 and ARC-AGI-3 are scored differently
ARC-AGI-2 is scored on correctness. A task is either solved or not, and the score is the fraction solved. That is why 24% on the private set is a meaningful number: roughly a quarter of the puzzles were answered exactly.
ARC-AGI-3 is scored on efficiency. Completing a level is the entry fee; the score is the human baseline action count divided by the agent’s, squared, capped at 1.15 and weighted by level number. An agent that finishes every level in ten times the human’s moves scores about 1% despite finishing everything. An agent that finishes nothing scores zero either way.
So a 24% on ARC-AGI-2 means “solved a quarter”. A 0.51% on ARC-AGI-3 does not mean “solved half a per cent”. It means that across the levels completed, the efficiency relative to a first time human was, on average, negligible. The two numbers are different quantities that happen to share a percent sign.
Why ARC-AGI-2 vs ARC-AGI-3 scores do not transfer
The ARC-AGI-3 report is explicit about the failure mode it was designed around. In a section titled “Overfitting and memorization shortcuts” it argues that “benchmarks that were designed to resist direct memorization can now be attacked via higher-level shortcuts if the public training set and the private test set are overly similar (e.g., identically distributed) and the model was trained on an enormous amount of tasks”.
That is a description of what happened to the static format. Given a public training set of input output puzzles, a system can be trained on enormous numbers of synthetic puzzles in the same distribution and learn the shape of the answer space without learning anything general. The ARC Prize 2025 report documents the technique that dominated that year: “the emergence of the refinement loop, a per-task iterative program optimization”, where a system generates candidate programs for each puzzle, tests them against the examples, and refines until one fits.
None of that transfers to an interactive game. There are no example pairs to fit a program against. The rules have to be discovered by acting, and the score punishes every action spent discovering them. A refinement loop that tries thousands of candidate programs per puzzle is, on ARC-AGI-3, an agent that takes thousands of actions per level, and the squared efficiency metric scores that at effectively zero.
What the refinement loop did to ARC-AGI-2, and cannot do to ARC-AGI-3
The 2025 report’s own framing is that the refinement loop was the story of its year, and that it raised a question about “knowledge-dependent overfitting”: whether the gains were general or were fitting to the distribution of the puzzles. ARC-AGI-3 is the answer to that question in benchmark form. The report says it will focus on systems “that have not been specially prepared for ARC-AGI-3, served behind a general-purpose API”, and routes harness based entries to a separate community board with an explicit caution against reading them as evidence of AGI progress.
The public set is deliberately unrepresentative for the same reason. The 25 downloadable games are, in the report’s words, “intentionally easier for both humans and AI” and do “not comprehensively represent the mechanics found in the private” sets. There is nothing to overfit to that would help on the private 110, because the private 110 were designed so that the public 25 would not.
The frontier scores since launch, 0.51 to 62.7 on the shared harness in five months, came from state carried between turns rather than from any refinement of the answer, which is the other way the two benchmarks differ: on ARC-AGI-3, what the system remembers across a game matters more than what it can compute about one frame.
Which benchmark to watch for a local model
ARC-AGI-2 remains the better test of a model’s raw one shot reasoning, and it is the one where a large local model can be compared to frontier systems on identical terms, because the task is a single completion. Nothing about it needs a harness.
ARC-AGI-3 is a test of agents, and the model is one component. The first milestone of its competition went to entries running 27B and 31B open weight models locally, which no ARC-AGI-2 leaderboard would have predicted, because what won was the wrapper: a Python console the model programs, or a rendered frame it reads and answers in JSON. A local model that cannot approach frontier ARC-AGI-2 numbers can still be the engine of a competitive ARC-AGI-3 agent, and the reverse is also true.
So do not read a system’s ARC-AGI-2 score as a forecast of its ARC-AGI-3 score, in either direction. One measures whether a model can infer a rule from examples. The other measures whether a system can find a rule by acting, and how few actions it wastes doing so. They are related the way an exam and an apprenticeship are related, and the second one is the one this year’s prize money is attached to.