A two level game played to a win in two actions returned an ARC-AGI-3 scorecard with a score of 100.0, a card id, and a per level breakdown, on a studio test machine with the network switched off. No API key, no server, no account. The scorecard is computed by the same package that plays the game.
That is not obvious from the documentation, which talks about scorecards as a server side object, so here is what the local one contains, how it is computed, and where it stops matching the one the leaderboard uses.
What arc.get_scorecard() returns in offline mode
After a few steps, arc.get_scorecard() returns an EnvironmentScorecard. Dumped to JSON it has a card_id, an overall score, and an environments list with one entry per game played. Each game entry carries its id in the id-version form and a list of runs, and each run records:
guid the wrapper instance that produced it
score the environment score for this run
levels_completed count of next_level() calls
actions total actions taken
resets how many RESETs were sent
state NOT_FINISHED, GAME_OVER or WIN
completed true when the game was won
level_scores one RHAE level score per level
level_actions actions spent on each level
level_baseline_actions the human baseline for each levelThe last three lines are the ones to keep. They are the arithmetic of the whole benchmark laid out per level: what the human needed, what you spent, and what that earned. For the 13 line toy game with baselines of one action per level and one action spent on each, level_scores read [100.0, 100.0] and the overall score was 100.0.
arc.close_scorecard() returns the same structure as a final document. The toolkit exposes five scorecard calls on the Arcade object, open_scorecard, create_scorecard, get_scorecard, close_scorecard and a scorecard_manager, and none of them needs the server in OFFLINE or NORMAL mode.
How the local ARC-AGI-3 scorecard computes a game score
The code is arc_agi/scorecard.py, and its EnvironmentScoreCalculator does three things. For each completed level it computes the human baseline divided by the actions taken, squared, times 100, and caps that at 115. It then averages the level scores using each level’s number as its weight, so the fifth level counts five times the first. Finally it caps the environment score at the weighted fraction of levels actually completed, so that finishing one level very efficiently cannot produce a high game score.
That is the formula from the methodology page, applied locally to the baseline_actions list that ships in each game’s metadata.json. We read the calculator against the technical report line by line, and the code caps the score rather than the ratio, which is the behaviour the prose describes and the printed equation does not. Whatever the server runs, this is what you get on disk, and it is the number quoted in the run entry’s score.
A level that was not completed contributes a zero at its weight, which drags the average down and also lowers the cap. An agent that completes three of five levels perfectly scores at most 6 out of 15, or 40.
Why a scorecard closes on its own after 15 minutes
The first line the toolkit logs on startup is the scorecard manager’s configuration: idle for 15 minutes, open for at most 3 days. A scorecard that receives no activity for a quarter of an hour is closed automatically, and one that stays busy is closed anyway at three days. Both limits are environment variables, STALE_MINUTES and MAX_OPEN_FOR_MINUTES, and the code clamps them to sane ranges rather than trusting whatever you set.
For a local run this rarely matters, because a run that has gone quiet for fifteen minutes has usually finished. For a local model that spends minutes on a single move it can matter a great deal. A long think that crosses the idle threshold closes the card under the agent, and the next action lands on a new one. If your agent is slow, raise STALE_MINUTES before you start, or open the card explicitly and check its id as you go.
How to read level_scores against level_baseline_actions
The three per level lists line up by index, and reading them together is the fastest post mortem an agent can get. Take a run that reports:
level_baseline_actions [22, 123, 73]
level_actions [44, 400, 0]
level_scores [25.0, 9.46, 0.0]Level one took double the human’s actions and kept a quarter of the score. Level two took 3.25 times the human and kept a tenth. Level three was never reached. The environment score is the level weighted average, (1 x 25 + 2 x 9.46 + 3 x 0) / 6, which is 7.3, and the completion cap is 3/6, or 50, so 7.3 stands.
Two things fall out of reading it this way. The level that dominates the score is the highest numbered one you failed, not the first one you passed. And a large resets count next to a modest actions count means the agent is dying and restarting rather than progressing, which is a harness problem before it is a model problem.
When the local ARC-AGI-3 scorecard differs from the server’s
The formula is the same, but three things are not.
The server’s scorecard is tied to the API key that opened it, so a registered key gives it a home and an anonymous key does not. The local one lives in the process and in whatever you dump it to. The scorecards page draws the line in one sentence: “Scorecards are not public, however you can share replays from scorecards created via the API with others. Local scorecards cannot be shared.” It also carries a warning worth taping to the monitor: “Stopping the program prematurely with Ctrl-C mid-run will not allow you to see the scorecard results.”
The server’s scorecard, in COMPETITION mode, cannot be read while it is open. The competition mode page lists that among the restrictions, along with a single scorecard per run and scoring “against all available environments, even if you choose not to interact with them.” Locally you can read the card whenever you like, and a game you never played simply is not on it.
And the server scores games you cannot download. The leaderboard numbers come from the semi private set, and the local scorecard can only ever cover the 25 public games on your disk. So the local card tells you exactly how your harness performs on the development set, in the benchmark’s own units, for free. What it cannot tell you is what the leaderboard would say, because the leaderboard is asking about games the card has never seen.