Grid to text for a local model: four lines of numpy, measured

The same 64 by 64 game board, sent to the same local model on the same machine, cost 442.7 seconds as raw text and 89.2 seconds as a four line summary. The model answered both times. The summary was 287 tokens; the raw grid was 4,179.

Grid to text is the whole problem for a local model playing anything on a board, and the cheapest conversion turned out to be the best one we measured. Here is the code and the numbers.

Why a raw grid is the worst grid to text conversion

The obvious encoding, one hex digit per cell, 64 rows of 64 characters, is faithful and terrible. It is about 4,200 tokens per frame before the model has said a word, and on gemma-4-31b-qat through LM Studio it triggered 4,274 tokens of reasoning before an answer, which at a 4,096 cap came back as an empty string four times in a row.

Compressing the text does not help, and that was the surprise. Run length encoding each row cut the prompt from 4,191 tokens to 1,522 and the model still needed the same 4,274 tokens of thought. It reasons over the cells you describe, not the characters you spend describing them. A 64 by 64 board is 4,096 cells however you spell it.

So the lever is not compression. It is telling the model less.

The four line colour summary, and what it costs

This is the summary, given a frame m as a numpy array of shape (64, 64):

def objects(m):
    lines = []
    for c in sorted(set(m.ravel().tolist())):
        ys, xs = np.where(m == c)
        lines.append(f"colour {c:x}: {len(ys)} cells, rows {ys.min()}-{ys.max()}, cols {xs.min()}-{xs.max()}")
    return "\n".join(lines)

For the opening frame of ls20 it produces nine lines, one per colour present, each with a cell count and a bounding box. That is 287 input tokens. The model thought for 990 tokens, terminated on its own, and named a legal action in 89.2 seconds.

What it throws away is shape. A bounding box does not say whether the colour is a wall, a corridor or scattered dots, and two objects of the same colour merge into one box. For the first move of a game that is often enough, because the first move is exploratory anyway. For a game that turns on the precise shape of a piece it is not, and the next format is the compromise.

How to downsample a grid with numpy without losing the picture

Block downsampling keeps geometry and drops resolution. Each k by k block becomes one cell holding the block’s most common colour:

def down(m, k):
    n = m.shape[0] // k
    return np.array([[np.bincount(m[i*k:(i+1)*k, j*k:(j+1)*k].ravel()).argmax()
                      for j in range(n)] for i in range(n)])

down(m, 4) turns 64 by 64 into 16 by 16, and down(m, 8) into 8 by 8. Rendered as hex rows, those cost 343 and 147 input tokens. On the same model and frame:

The reasoning cost climbs with cells, not tokens, and it climbs sublinearly: 64 cells cost 692 tokens of thought, 4,096 cost 4,274. The 16 by 16 is the practical middle. It keeps every object larger than a 4 by 4 block, which on an ls20 frame is all of them, because the game’s own camera is 16 by 16 and the engine scales it up by four to fill the frame. Downsampling by four is not lossy for that game at all; it is undoing the upscale.

That is worth checking per game. The technical report fixes the output at 64 by 64, but the camera behind it is whatever the game declares, and a game drawn at 32 by 32 loses detail at k=4 that it kept at k=2.

What the summary loses, and when it matters

Both formats destroy something, and the right question is whether the destroyed thing was needed for the next move.

The colour summary loses shape and merges same colour objects. It also loses position within a bounding box, so “the player is somewhere in rows 20 to 45” is all it can say. It is the right format for the first few moves of an unknown game, where the agent is finding out what changes when it presses a key, and for any game where the pieces are big and distinct.

The downsample loses anything smaller than a block, which includes one pixel markers, thin walls and the exact edge of a region. It keeps layout, which is what a model needs to reason about paths and adjacency.

Neither loses colour identity, and that is the thing to protect. Every mechanic in these games is expressed through colour changes, and a format that collapses colours to save tokens, for example by mapping them to a few symbols, throws away the only channel the game uses to communicate.

Which grid to text format to use for which job

Use the colour summary to orient. It is cheapest, it terminates fastest, and it gives the model a list of things rather than a wall of digits. Send it with the legal actions and ask for one move.

Use the 16 by 16 downsample to play. It is the coarsest format that keeps the layout, and it costs about a third of the full frame’s thinking time. Ask for structured output while you are at it: with a JSON schema that only permits the four action names, the same 16 by 16 frame took 1,078 tokens of reasoning instead of 2,136 and answered in 94.7 seconds, which was the shortest run of the session, though a single one, and identical prompts on this stack have varied by more than that. LM Studio’s structured output takes the schema in response_format.

If the model can see, send the board as a picture: the same frame cost 187 prompt tokens as a PNG against 4,179 as hex. Send the raw grid only when a move genuinely turns on a single cell, and set the token cap far above 4,274 when you do, because a cap that is 4% too low returns nothing and looks like a broken model.

And measure on your own hardware. Every number above is one model, one machine, one frame, and the shape of the result is more transferable than the seconds. What transfers is this: a local model pays for what it has to look at, and four lines of numpy decide how much that is. The first agent only needs the summary to start.