Gemma 4 31B has run on our test machine at 65,536 tokens of context for months, served by LM Studio. When Hermes Agent’s own engine loaded the same file on 12 September, it gave the model 193,280 tokens per slot across four slots, with nobody asking for either number. This piece traces where that figure comes from when you run Hermes Agent with Gemma 4 through the built-in Local Models engine, what Hermes’s memory estimator gets wrong about the model, and the setting that avoids it on a 48 GB Mac.
The evidence is Hermes’s own llama-server log, its launch settings file and its source code, read on an M4 Pro with 48 GB of unified memory.
Why Hermes Agent loads Gemma 4 31B at 193,280 tokens of context
Gemma 4 31B is not in Hermes’s model catalog. We added it with the Add model file button, which hard-links a GGUF already on the Mac, so Hermes and LM Studio pointed at one 17.6 GB file on disk.
For the models it ships, Hermes writes launch settings into a file called presets.ini: a context window, KV cache type and batch sizes. On our machine that file was 0 bytes. With no entry for Gemma, Hermes’s router started the model with this command line, copied from ~/.hermes/logs/llama-server.log:
llama-server --host ... --jinja --metrics --port ... --slots --no-webui
--alias gemma-4-31B-it-QAT-Q4_0 --direct-io
--model ~/.hermes/models/gemma-4-31B-it-QAT-Q4_0.ggufThree flags are missing, and llama.cpp fills each gap with a default. Its server documentation lists --ctx-size as “default: 0, 0 = loaded from model”, so with no context set, the server takes the size from the model file. --parallel defaults to “-1 = auto”, and a unified KV buffer is “enabled if number of slots is auto”. The log line that followed shows the result:
initializing, n_slots = 4, n_ctx_slot = 193280, kv_unified = 'true'The same line appears for three separate loads in the log. Google’s Gemma 4 31B model card gives the model a 256K context window, so 193,280 is below the maximum. We have not traced why llama.cpp settled on that exact figure. What matters here is that no part of Hermes chose it.
The third missing flag is --mmproj. In llama.cpp that option is the path to the multimodal projector, the file that lets a model take images. Hermes passed none for Gemma, so this load had no projector to read an image with, although the Gemma 4 model card lists image input.
What Hermes’s memory estimator gets wrong about Gemma 4
The empty settings file has a cause. Before writing launch settings, Hermes estimates how much memory each model needs at its minimum 64K window, and for Gemma 4 the estimate was far too high. Its preset generator logged a refusal: the model “needs ~72.2 GiB at the 64K floor but only ~38.4 GiB of VRAM+RAM exist”, followed by advice to try a smaller quant.
The number comes from one line in hermes_cli/local_runtime/estimator.py. It holds a table of how many layers use sliding-window attention in each Gemma generation, and it lists gemma3 and gemma2 only. A sliding-window layer keeps a short, fixed stretch of context in memory instead of the whole conversation, so it costs far less KV cache. Gemma 4 mixes both kinds; the model card describes “a hybrid attention mechanism that interleaves local sliding window attention with full global attention”. Without a table entry, Hermes prices all 60 layers as full attention.
The estimate is wrong by a wide margin. On 11 September we started Hermes’s own llama-server build with Hermes’s settings for dense models and the context pinned to 65,536. Gemma loaded in 11.3 seconds and reported n_ctx_slot = 65536, and it answered 11 of 12 ARC-AGI-3 board prompts in our Qwen3.6 vs Gemma 4 test. LM Studio has held the same file at that window all along.
So the refusal did not stop Gemma from loading. It stopped Hermes from writing the one setting that would have capped the window. Meanwhile the Local Models page listed the file as “Added by you” with a blue Use button and no warning.
Hermes Agent and LM Studio on one 48 GB Mac: running both copies of Gemma 4
On our machine LM Studio keeps Gemma 4 31B loaded at 65,536 tokens for our assistant. Measured on 10 September, that load used about 10 GB of process memory on top of the 17.6 GB of weights, which LM Studio maps from disk.
When Hermes’s engine then loaded the same model at 193,280 tokens per slot, the Mac held two servers for one model: one sized for 65,536 tokens and one sized from the file’s own metadata. A 64K window alone costs about 4.4 GB of memory beyond the weights, and the KV cache grows with the window, so a cache sized for 193,280 tokens per slot is several times the one LM Studio already held. We did not keep a memory reading from that load, so we give no figure for it. Hermes’s config backup is timestamped 12:19 on 12 September, the moment we switched Hermes back to LM Studio.
The --direct-io flag makes this harder on memory. The llama.cpp documentation describes the default load mode as memory-mapping the model file, and dio as “use DirectIO if available”. A memory-mapped model can be dropped and re-read from disk when the system needs room. Weights read in with direct I/O sit in the server’s own memory instead. The llama.cpp build Hermes ships also prints a warning that --direct-io is deprecated in favour of --load-mode dio.
We ran into the same pattern when we ran two models at once on 48 GB: each one fits on its own, and the failure comes from what the second one assumes about the memory the first one already holds.
How to run Gemma 4 in Hermes Agent with a fixed context window
The setting that avoids all of this is to let one engine own the model. Our assistant now runs Gemma 4 31B through LM Studio, and the Hermes config reads:
model:
base_url: http://127.0.0.1:1234/v1/
default: google/gemma-4-31b-qat
provider: lmstudio
local_runtime:
enabled: falseWith local_runtime.enabled set to false, Hermes starts no llama-server, and the context is whatever LM Studio loaded. The full walkthrough is in our guide to Hermes Agent local models on a Mac. Hermes reads the file when its gateway starts, so the change applies after a gateway restart. Hermes’s Local Models documentation supports this route directly: “Point a custom endpoint at any OpenAI-compatible server for full manual control”.
LM Studio has its own trap on context length. A model loaded on demand through its API starts at 8,192 tokens unless told otherwise, which we covered in why a local model stops at exactly 8,192 tokens. Load the model explicitly with a context length before Hermes connects.
If you would rather keep Hermes’s engine, the constraint is simple: do not load the same large model in LM Studio at the same time on a 48 GB machine. Check the log after the first load, since the n_ctx_slot line is the only place the chosen window is recorded.
Hermes’s documentation says every recommended model gets at least a 64K window and grows toward its native maximum as a conversation needs more room. For the catalog models that is a policy with limits. For a model you add yourself, the Gemma 4 case shows the window comes from the file and the slot count comes from llama.cpp, and nothing in the Hermes window shows either number.