A random agent, pressing legal keys with no idea what it was doing, finished level 1 of an ARC-AGI-3 game. It took 40,699 actions. A person who had never seen the game before takes 22.
Feeding both numbers to the benchmark’s own scoring code returns a level score of 0.0000292198 out of 100. Not a failing grade. Not a low one. Effectively nothing, for a level that was genuinely completed.
That is RHAE working exactly as designed, and it is why ARC-AGI-3 scores look so strange next to every other benchmark you have read about this year.
What RHAE measures, and why it is not accuracy
RHAE stands for Relative Human Action Efficiency. Almost every benchmark you know reports accuracy: how many items did the system get right, out of how many. RHAE does not ask whether you finished. It asks how many moves you needed compared to a person who was seeing the game for the first time, and it treats that ratio as the entire score.
The distinction matters because ARC-AGI-3 is a set of interactive games rather than a set of questions. In a game you can always finish eventually. Press enough buttons and something will happen. A pass or fail metric would reward the agent with the biggest budget, which is the one thing the benchmark is built to resist.
It is also why the saturation claims of the last fortnight do not survive inspection: a perfect score on games whose source you can read says nothing about efficiency on games you cannot.
So completion is the entry fee, not the score. The score is efficiency, and the reference point is a human who was as ignorant as your agent was at the start.
How much the RHAE formula charges for extra actions
The per level formula is a squared ratio. Let h be the human baseline action count for that level and a be the actions your agent took:
level_score = min( (h / a)^2 * 100, 115 )Squaring is the whole design. A linear metric would let you buy a mediocre score with a bigger action budget. A squared one makes that trade terrible, fast.
| Actions taken | As a multiple of the human | Level score |
|---|---|---|
| 22 | 1x | 100.00 |
| 33 | 1.5x | 44.44 |
| 44 | 2x | 25.00 |
| 66 | 3x | 11.11 |
| 110 | 5x | 4.00 |
| 220 | 10x | 1.00 |
| 40,699 | 1,850x | 0.0000292198 |
The RHAE power law. Scores computed with the toolkit’s own scorecard against the ls20 level 1 baseline of 22 actions, 4 September 2026.
Take fifty per cent more moves than the person did and you have already lost more than half the score. Take double and you keep a quarter. Take ten times as many, which is still a wildly competent agent by any normal standard, and you keep one point in a hundred.
The last row is our own random agent. It cleared the level. It scored nothing.
Level scores then roll up into an environment score as a weighted average, using the level number as the weight, so level 5 of five counts five times as much as level 1 and contributes 5/15ths of the total. Tutorial levels barely register. The environment score is then capped at the weighted fraction of levels you actually completed, so clearing one easy level of seven cannot be inflated by being efficient at it. Running our random agent’s single completed level through the real calculator, with the other six left unfinished, returns an environment score of 0.0000010436.
Where the ARC-AGI-3 human baseline comes from
The baseline is not the best human, and that is the most commonly mangled part of this metric.
ARC Prize’s methodology page defines it as the upper median human by fewest actions, across multiple first time players, and spells out the arithmetic: “For an even number of players, the upper of the two middle entries is selected. For example, if four players complete a level, third place is the baseline; if five players complete it, third place is still the baseline.”
Third place out of four or five. Not the winner, and not the average either. The choice filters out the person who got lucky while staying anchored to real play rather than a theoretical optimum. The technical report puts the sample behind this at 486 participants across 414 candidate environments and 2,893 attempts, with a median attempt lasting 7.4 minutes. Each candidate environment was tried by ten people, and it was only admitted if at least two of them solved it outright, all levels, on first sight.
You do not need an API to see these numbers. They ride along with the game. Install the toolkit, pull a game, and the baseline is sitting in metadata.json as a plain list:
"baseline_actions": [22, 123, 73, 84, 96, 192, 186]Seven levels of ls20, and the number of moves a middling first time human needed for each. Your agent’s score is that list divided by whatever your agent managed, squared.
Why the RHAE formula in the paper does not match the code
Here is something worth knowing before you cite a number: the equation printed in the technical report and the equation the toolkit actually runs are not the same equation.
The report gives the level score as min(1.15, h/a)^2. That caps the ratio at 1.15 before squaring it, so the highest possible level score is 1.15 squared, or 1.3225.
The prose two pages later says something different. It describes capping “the maximum score for a level at 1.15x the human baseline”, and the methodology page repeats that: an agent that finds a shortcut “can receive at most 1.15”.
The shipped code settles it. In arc_agi/scorecard.py, the calculator does this:
score = ((baseline_actions / actions_taken) ** 2) * 100
score = min(score, 115.0) # Cap at 115It squares first and caps the score afterwards, which matches the prose and not the equation. The comment one line above still reads “max 100”, which is stale by 15 points.
The two only disagree when an agent beats the human baseline, and running both against the shipped calculator shows exactly where:
| Actions, as a fraction of the human baseline | Shipped code | Report equation |
|---|---|---|
| 0.93x (20 actions against 22) | 115.00 | 121.00 |
| 0.87x (19 actions) | 115.00 | 132.25 |
| 0.50x (11 actions) | 115.00 | 132.25 |
| 1x and slower | identical | identical |
Nobody scoring below human parity will ever notice, which is currently everybody. It will matter the moment an agent starts beating the baseline on individual levels, and by then the difference is a leaderboard position. If you are computing RHAE yourself, use the code, not the paper.
What RHAE means if you are running an agent locally
The metric changes what is worth optimising, and it does it in a way that happens to suit a small local model better than you would expect.
Under an accuracy metric, more attempts are free. Under RHAE they are the score. That kills the entire class of approach where you loop a cheap model a few thousand times and take whatever works, which is exactly the approach a fast local model invites. Our random agent is the extreme version of that strategy and it earned 0.0000292198.
What it rewards instead is an agent that works out the mechanics and then acts deliberately, which is a thinking problem rather than a throughput problem. That is why the first ARC-AGI-3 milestone was won by a 27B model running locally rather than by whoever could afford the most API calls, and why the second and third place entries used a 31B, the same class of model we benchmarked on this machine earlier this year. Tokens per second stop being the bottleneck when the budget you are spending is measured in moves.
The practical warning is the flip side. If a single move costs your local model several minutes, as ours does when handed a large enough board, then the 22 move human baseline for one level of one game is already an hour and a half of your machine. RHAE will not punish you for being slow in seconds. It will punish you for every extra move you spend finding out what the game wants.
Every benchmark encodes an opinion about what intelligence is for. Accuracy metrics say it is for being right. RHAE says it is for working things out quickly, and it charges the square of your inefficiency to make the point. The number that will decide ARC Prize 2026 is not how many games an agent can finish. It is how few moves it wastes learning how.