The second time you run a benchmark it gets faster, and everyone knows to throw away the first result as a warm-up. On a local model that instinct is exactly backwards. We spent an evening measuring prompt processing speed on a 31B model and got 13,763 tokens a second, a number so good it should have been the story. It was wrong by a factor of 182, and the reason is that we had thrown away the only honest measurement in the set.
Everything below was measured on a studio work/test stack, an M4 Pro with 48 GB of unified memory, running gemma-4-31B-it-QAT at Q4_0 through LM Studio’s llama.cpp Metal backend. The scripts are in the workspace and the raw numbers are in the article.
Why does the same prompt run 182 times faster the second time?
The first script did what benchmark scripts do. Build a prompt of a given length, send it three times, discard the first as warm-up, take the median of the rest. Measure time to first token, which is the moment prompt processing ends and generation begins.
The result looked like a hardware win. Time to first token was flat at about 0.43 seconds no matter how long the prompt was, from 1,752 tokens up to 5,996. The implied processing rate climbed from 4,244 tokens a second to 13,763, and the spread between the fastest and slowest sample was a few milliseconds. Clean data, tight error bars, completely false.
The tell arrived from a different test. A 25,407-token prompt sent once, as part of a memory measurement, took 254 seconds. If prompt processing really ran at 13,763 tokens a second, that prompt should have taken under two seconds. Two measurements on the same machine, the same model and the same afternoon disagreed by two orders of magnitude, so at least one of them was garbage.
The one that was garbage was ours. llama.cpp keeps the KV cache from the previous request and reuses whatever prefix of the new prompt matches it. Send an identical prompt a second time and the prefix match is total: there is nothing left to process, and time to first token collapses to the cost of a single decode step. Discarding the first run and keeping the rest is a procedure that discards every cache miss and averages the hits.
What is the real prompt processing speed on an M4 Pro?
We rewrote the probe to put a random nonce at the front and back of every prompt, so no two requests share a prefix and the cache can never serve one. Same model, same machine, same context setting, three samples each.
| Prompt tokens | Time to first token | Processing rate |
|---|---|---|
| 108 | 2.12 s | 51 tok/s |
| 761 | 11.08 s | 69 tok/s |
| 1,813 | 27.87 s | 65 tok/s |
| 3,565 | 53.98 s | 66 tok/s |
| 5,197 | 79.56 s | 65 tok/s |
Prompt processing on this model and this machine runs at about 65 tokens a second, and it scales linearly with prompt length. There is no flat region and no free lunch. The 108-token row is slower per token because a fixed overhead of roughly a second is being spread over very few tokens, which is the only thing the first script got right.
Set the two side by side at a comparable prompt length. Cached, a roughly 6,000-token prompt reported 0.436 seconds. Uncached, a 5,197-token prompt took 79.56 seconds. That is 182 times slower on a shorter prompt.
Where does the time actually go in a local model reply?
With prefill measured honestly, the split changes completely. Generation on the same model runs at 10.3 to 11.5 tokens a second, and a reply capped at 300 tokens takes 19 to 28 seconds. That decode rate is the half that speculative decoding is meant to attack, which is worth knowing before you spend a download on a draft model to fix the wrong bottleneck.
So for a 5,200-token prompt with a 290-token answer, prompt processing costs about 80 seconds and generation costs about 27. Prompt processing is roughly 75 percent of the wait. Our cached run put it at 1.5 percent. Anyone tuning a local setup on the cached number would optimise generation, which is the smaller half of a problem they have measured upside down.
There is a second thing eating the generation half, and it is only visible if you count what the model streams rather than what it shows you. gemma-4-31B-it-QAT is a reasoning model, and it emits reasoning_content deltas before any content. Across our runs it produced 254 to 297 thinking tokens against 35 to 36 visible ones. Roughly 87 percent of the tokens it generated, and 87 percent of the generation time, went to text the user never reads. At the shortest prompt in the set it spent all 297 tokens of its budget thinking and returned no answer at all, because the cap landed before it finished.
That reframes the usual advice about context. The cost of a long prompt is real and linear, and the cost of a reasoning model is a multiplier on top of the half everyone already worries about. Both are worth knowing before you size a machine, which is the same reason we went through the tensor tables in the Nemotron GGUF audit rather than trusting a file listing.
How do you benchmark a local model without measuring your own cache?
Four habits fix this, and they cost nothing.
Vary every prompt. A random string at the very start of the prompt is enough, because prefix matching fails at the first differing token. A nonce at the end will not do it: everything before the difference still matches and still gets reused.
Keep the first result. On a cold cache the first run is the only honest one in a repeated set. If you want a warm-up, warm the model with a different prompt, then measure once.
Report prompt processing and generation as separate numbers. A single tokens-a-second figure hides which half you are actually paying for, and on this machine the two differ by a factor of six in the model’s favour on generation.
Sanity-check against a single long request. A one-off prompt several times larger than your test set should take proportionally longer. When it does not, something is being reused. That mismatch is what caught this one, and it was luck rather than method.
None of this makes the cache a problem. Prefix reuse is genuinely useful, and a coding agent that keeps a long system prompt stable across turns gets most of it back for free on every request after the first. The problem is only that a benchmark which repeats one prompt measures that best case and reports it as the model’s speed. The number to publish is the cold one, because the cold one is what a new conversation costs.