Twenty random actions on ls20, recorded, produced a 282,186 byte file on a studio test machine. The same twenty actions recorded without frame data produced 7,100 bytes. At 60,000 actions, the budget our random baseline uses per game, the first setting would write roughly 770 megabytes per game and the second about 19.
ARC-AGI-3 recordings are one keyword argument away, and the only decision that matters is whether the frames go in. Here is what the file contains either way, and how to read it back.
How to turn on ARC-AGI-3 recordings in the toolkit
One argument to make():
env = arc.make("ls20", save_recording=True)The recordings page confirms that is the whole switch for the Python toolkit, and adds that runs made through the API or the benchmarking harness are recorded automatically on their side. Locally nothing is recorded unless you ask.
The file lands under recordings/, or wherever recordings_dir points, in a directory named after the scorecard and a file named after the game and the wrapper instance. On this machine that was:
recordings/d36c3676-c0e8-4322-8e85-5ee85a89742b/ls20-9607627b-c899d718-08a4-4498-a44e-aba7165d8d52.jsonlThe docs give the pattern as recordings/<scorecard_id>/<game_id>-<guid>.jsonl, and in practice the game id is the versioned form, ls20-9607627b, so a recording carries the exact version of the game it was made against. That matters for the same reason version pinning does: a replay against a revised game is not a replay.
Recording works in every mode. The file above was written in OFFLINE mode with no key and no network, and it is the same format the online harness writes.
What one line of an ARC-AGI-3 recording contains
The file is JSON Lines, one record per action, including the initial RESET. Twenty actions plus a reset gave 22 lines, so there is one line of overhead. Each line is an object with two keys, timestamp and data, and data is the frame the wrapper returned for that action:
action_input {"id": "RESET", "data": {}, "reasoning": null}
available_actions [1, 2, 3, 4]
frame a list of 64 by 64 grids, one per rendered step
full_reset whether this reset rebuilt every level
game_id "ls20-9607627b"
guid the wrapper instance
levels_completed 0
state "NOT_FINISHED"
win_levels 7Two of those repay attention. action_input carries the reasoning field, which is the toolkit’s slot for whatever your agent wants to attach to a move, echoed back verbatim up to 16 KB. If your harness writes the model’s explanation there, the recording becomes a transcript of why each move was made, not only what it was. And frame is a list, because one action can render several frames when a game animates between turns; a recording keeps all of them.
There is no score on a line. Scores live on the scorecard, which the recording’s directory is named after, and the scorecards page is blunt about what happens if you interrupt: “Stopping the program prematurely with Ctrl-C mid-run will not allow you to see the scorecard results.” The recording survives an interruption. The card may not.
How big ARC-AGI-3 recordings get, with and without frames
With frames, the median line was 12,827 bytes, almost all of it the 4,096 cell grid written out as JSON integers. The arithmetic is plain: four thousand cells at two or three characters each, a digit plus a comma and a space, is about 12 KB before a single field of metadata, and an action that renders more than one frame multiplies that again. Without frames, set with include_frame_data=False on the same make() call, the median line was 322 bytes and the frame key is simply absent.
That is a factor of forty, and it decides what recording is for. At 60,000 actions per game and 25 games, a full recording of the random baseline would be on the order of 19 gigabytes; the frameless version, under half a gigabyte. For an agent that takes a few hundred actions per game the full version is a few megabytes and there is no reason not to keep it.
The frameless file still has everything except the picture: every action, every state change, every level completion, and the reasoning blob if you set one. Since the games are deterministic given the same version, the frames can be regenerated by replaying the actions, which is the next section.
How to replay an ARC-AGI-3 recording locally
The toolkit’s replay is a web page. For games played through the API, the docs point to https://arcprize.org/scorecards/<scorecard_id>, where the scorecard’s recordings can be stepped through and, per the scorecards page, shared. Local recordings have no such page, and the docs say so: “Local scorecards cannot be shared.”
Locally, the file is enough on its own. Reading it back is a loop over JSON lines, and re-rendering a frame is numpy.array(record["data"]["frame"][-1]), which gives the 64 by 64 grid the agent saw after that action. For a frameless recording, replay the action_input sequence through a fresh make() of the same versioned game in offline mode and the frames come back identical, because the engine rebuilds levels from clean copies on every reset and a game’s behaviour depends only on the actions it receives.
That second path is worth setting up once. It turns a 322 byte per action log into a complete, inspectable run, and it lets you diff two agents’ recordings action by action to find the move where they diverged.
When to record, and when not to
Record every run of an agent that has a model in it. At a minute or more per move, a run is expensive to reproduce, and the recording is the only artefact that lets you look at a bad decision after the fact without paying for it again. Keep frames on; at that action rate the file is small.
Do not record a random or a search agent with frames on. Sixty thousand actions at 12.8 KB each is a file nobody will open. Record frameless, or not at all, and regenerate what you need from the action list.
And read the file at least once by hand. The record for a RESET shows full_reset true or false, which tells you which of the two kinds of reset the engine performed; the levels_completed field on consecutive lines shows exactly when a level fell; and the available_actions list changing between lines is the earliest sign that a game has more than one phase. None of that is visible from the scorecard, and all of it is in a file the toolkit writes for the cost of one argument.