Nobody writes a system prompt expecting it to be expensive. It is a text file, it is measured in kilobytes, and kilobytes have not been a unit anyone worried about for twenty years. Then you count the agent system prompt tokens in one and discover that a single 75 KB markdown file is 20,007 tokens, which does not fit in the 8,192-token window the model was loaded with, and would take five minutes to process if it did.
We counted four real files: the ones this studio’s agents actually load. The count came from the tokenizer of the model being run, gemma-4-31B-it-QAT, by reading usage.prompt_tokens off the completions endpoint rather than estimating from characters.
How many tokens is an agent system prompt, really?
| File | Size | Tokens | Bytes per token |
|---|---|---|---|
SOUL.md, the agent’s persona | 5,085 B | 1,230 | 4.1 |
AGENTS.md, the workspace guide | 7,059 B | 1,897 | 3.7 |
hermes-agent/AGENTS.md | 75,142 B | 20,007 | 3.8 |
| skills prompt snapshot, JSON | 47,098 B | 15,788 | 3.0 |
The rule of thumb that four bytes make a token holds for prose markdown and breaks for structured data. The JSON skills manifest packs a token into every three bytes, because punctuation, braces and quoted keys tokenise far denser than English does. If you estimate a JSON context file at four bytes per token you will be about 25 percent under.
Added together the four files are 38,922 tokens. That is 475 percent of an 8,192-token context window, and 119 percent of a 32,768-token one. The full context an agent might load does not fit in a 32K window at all, before the user has typed a character or the agent has read a single file.
The two large files proved this the hard way. At 8K context both returned HTTP 400 rather than a completion, because the request could not be tokenised into the window. We reloaded the model at 32K to get their counts, and even then they fit only individually, never together.
What do agent system prompt tokens cost in RAM and time?
Two numbers we measured on the same machine this week turn the token count into a bill.
The first is memory. The KV cache on this model costs 80 KB per token, a figure that comes out of the architecture and matched the measured resident set to the kilobyte. At that rate SOUL.md occupies 96 MB, the workspace guide 148 MB, the large agent file 1,563 MB, and the skills manifest 1,233 MB. All four together are 3.0 GB of RAM, spent before any work happens.
The second is time. Cold prompt processing on this machine runs at about 65 tokens a second, and this measurement confirmed it independently: the 20,007-token file took 329.1 seconds and the 15,788-token file took 260.7, which works out at 60.8 and 60.6 tokens a second. Three separate experiments now agree on that rate.
So the four files together are about 599 seconds of prompt processing, just under ten minutes, on a cold cache. That is not a figure anyone would accept if a tool announced it up front, and it is exactly what a fresh agent session with a full context load costs on this hardware.
Why does the cache make this survivable anyway?
Because a system prompt is the best possible cache candidate, and this is the one place where the prefix reuse that ruined our earlier benchmark works entirely in your favour.
llama.cpp reuses the KV cache for any matching prompt prefix. A system prompt sits at the very front of every request in a session and does not change between turns, so after the first message the whole thing is a prefix hit. The ten minutes is paid once per session, not once per turn. That is why agents with enormous context files feel usable in practice despite arithmetic that says they should not.
The corollary is where it hurts. Anything that changes the front of the prompt invalidates everything after it. Injecting a timestamp, a rotating session id, or a freshly reordered tool list at the top of the system prompt turns a free prefix into a full reprocess on every single turn. On these numbers that is the difference between paying 599 seconds once and paying it repeatedly. Put volatile content at the end of the prompt, never the beginning.
What should you actually do about a large system prompt?
Measure it first, with the tokenizer of the model you run rather than a character estimate, because the two disagree most on exactly the structured files that tend to be largest.
Then decide what genuinely needs to be resident. A 20,007-token file that documents every command of a tool the agent uses twice a week is 1.5 GB of KV cache held for the whole session. The same content in a file the agent reads on demand costs nothing until the moment it is needed, and the read itself is a few hundred tokens. The question to ask of each block is not whether it is useful, it is whether it is useful on *every* turn, because that is what loading it into the system prompt actually buys.
Watch the interaction with everything else you have loaded, too. Three gigabytes of KV cache for context files is roughly what a second model resident alongside the first costs on this machine. Those are real alternatives competing for the same 48 GB, and the context files are the ones nobody puts on the sizing spreadsheet.
Two caveats on the counts themselves. They are gemma-4’s tokenizer, and a model with a different vocabulary will produce a different number for the same file, usually within about 10 percent for English prose but further out for dense JSON, so re-count rather than carry these figures across to another model. And these are four files from one studio, chosen because they are the ones actually loaded here, not a survey of what agent context files look like in general. The method transfers even where the numbers do not: read usage.prompt_tokens back from the model you run, on the file you actually load.
The uncomfortable part is that none of these files looked large. The biggest is 75 KB, which is smaller than a phone photograph, and it is a fifth of a 100K context window on its own. Kilobytes stopped being the unit that matters the moment the thing reading them started charging 80 KB of memory per token.