A MoE is five times faster and bigger in memory

What does the A in 26B A4B mean for memory and speed?

Google states the convention plainly on the Gemma 4 model card: “the ‘A’ in 26B A4B stands for active parameters.” The same card gives Gemma 4 26B-A4B a 256K context window. Google DeepMind released it on 2 April 2026 under the Apache 2.0 licence.

Every expert must be resident, because the router can select any of them at any token. Only the routed ones do arithmetic. A model labelled “3B active” therefore still needs memory for every parameter it has, 35 billion in the case of Qwen3.5-35B-A3B.

The rule holds at the largest scale. Qwen3.8-Max, whose weights Alibaba published in August, activates 95 billion of its 2.4 trillion parameters per token, and all 2.4 trillion have to be held in memory.

The architectures now shipping locally:

Llama 4 uses a different convention (17Bx16E means 17B active across 16 experts) and Meta routes each token to “the shared expert and also to one of the 128 routed experts”, i.e. top-1, unusually sparse.

How much faster is a MoE than a dense model on a Mac?

One comparison of the two designs on the same Mac appears in an official project record. Ollama issue #15368, filed in April against Ollama 0.20.2 on an M5 Max with 128 GB of unified memory, reports that Ollama’s llama.cpp runner reached about 75 tok/s on Gemma 4 26B-A4B and about 15 tok/s on the 31B dense model.

That is roughly a fivefold gap on an eightfold cut in active parameters (3.8B against 30.7B), which is about what the bandwidth argument below predicts. Treat it as one data point: the figures are rounded, and the report was written while debugging a flash-attention hang, not as a benchmark.

Why: bandwidth, and only bandwidth

Apple Silicon is bandwidth-bound on dense decode. Apple’s specifications put the M4 Pro at 273 GB/s of memory bandwidth, half the M4 Max’s 546 GB/s, and on StrideNote’s M4 Pro a dense Gemma 4 31B generates a median of 11.0 tokens a second. The rest of our measurements on this machine are in our M4 Pro benchmarks.

MoE attacks exactly that. Fewer weights streamed per token means less to read, and reading is the constraint.

We expected a second effect and were wrong about it. Our working assumption was that immature expert-routing kernels on Metal and MLX would eat part of the theoretical win. The source code says otherwise, and the correction is more interesting than the guess.

In llama.cpp’s ggml-metal-ops.cpp:

// ne21 = n_rows (batch size)
const int ne21_mm_id_min = 32;
if (props_dev->has_simdgroup_mm && ne00 >= 64 && (ne21 >= ne21_mm_id_min)) {
    // kernel_mul_mm_id_map0 (expert id-map gather) + kernel_mul_mm_id
} else {
    // kernel_mul_mv_id
}

Below batch 32, which is all decode, Metal never runs the expert gather path at all. It uses a plain per-expert matrix-vector kernel. MLX has the same shape of threshold: GatherQMM::eval_gpu requires M == 1 && B >= 16 && right_sorted_ && B / E >= 4, conditions a single decode token cannot meet for top-8-of-128, and mlx_lm/switch_layers.py sorts only when indices.size >= 64.

So expert-gather cost is structurally a prefill concern. Every published gather optimisation confirms it. llama.cpp PR #12612 is the cleanest datapoint: DeepSeek V2 Lite prefill went 150.94 to 968.34 t/s, a 6.42x gain, while decode moved 94.87 to 97.81 t/s, 1.03x. PRs #13388 and #15541 are the same story at 2.21x and 1.58x prefill.

MoE decode on Metal is memory-bandwidth-bound, and there is corroborating evidence in a place you would not look for it: MTP speculative decoding loses on Metal. llama.cpp issue #23752 reports Qwen3.5-9B-Q4_K_M dropping from 25.3 tok/s baseline to 19.3 at n_max 6, because the extra weight reads cost more than the saved steps. That is the signature of a bandwidth-bound system.

It also suggests a limit worth testing before you rely on it: if you already run MoE, MTP may have less left to give you, because two optimisations chasing the same bottleneck do not stack. No official project record we could find measures MTP on a MoE and a dense model side by side.

One source to treat carefully: arXiv 2604.18788 (NPUMoE) does say “expert routing is unpredictable and introduces dynamic tensor shapes”, claiming 1.32-5.55x latency gains. But that is the Apple Neural Engine, not the Metal GPU, a different machine with static-shape constraints the GPU does not have. It is not evidence about MLX or llama.cpp.

Why can a MoE need more memory than a dense model?

This is where the two-number rule bites.

These are download sizes from Ollama’s official model library (qwen3.5, gemma4), the weights alone before any KV cache.

The Qwen MoE is larger than the dense model beside it, even though it computes with a ninth of the active parameters. There is no contradiction: footprint tracks total parameters, so a 35B MoE outweighs a 27B dense, and a 25.2B MoE undercuts a 30.7B dense. Speed tracks active parameters regardless.

The practical rule: size your RAM against the total, and your speed expectations against the active count. A “3B active” model is not a 3B model on your memory budget.

MoE does not shrink the KV cache, and this is the part that decides long-context work. Attention layers are untouched by expert sparsity, so expert count is irrelevant here. What matters is the attention configuration, and it varies sharply. Qwen3.5-35B-A3B runs 3 GatedDeltaNet layers to 1 full attention, replacing a growing KV cache with fixed-size recurrent state in 75% of layers. Gemma 4 26B-A4B runs 25 sliding-window to 5 full attention across 40 layers.

At long context the cache dominates the weights outright. SharpAI’s SwiftLM project reports Gemma 4 26B on an M5 Pro at 40K context needing 54.8 GB with an FP16 KV cache, falling to 23.9 GB with quantized KV; at 100K, 54.3 GB falling to 26.4 GB. That matches what we saw ourselves with dense models in our tool-calling work, where Gemma 4 31B at 64K context took about 35 GB and generated at roughly 2 tok/s.

What if it does not fit? Expert offloading

Because only a few experts fire per token, you can keep most of them off the GPU. Nothing is merged in either project, but the published results are striking.

The clearest record is llama.cpp PR #23440 (open, not merged), on Qwen3-30B-A3B-Q6_K, M3 Pro 36GB: vanilla 38.1 tok/s; 80 slots 29.1 tok/s at ~18.9 GiB wired (>90% hit rate, ~8.3 GiB saved); 16 slots 15.7 tok/s at ~7.8 GiB. Also 13 tok/s on an M1 Pro 16GB.

Note what that trade actually is: residency buys capacity, not speed. mlx-lm PR #1588 (closed without merging) states the cost plainly: when the model already fits, offloading is “5x slower decode and 8 to 12x slower prefill.”

What does a MoE give up in quality?

Both vendors publish MoE against dense in the same table, which is unusually honest and worth using.

Gemma 4 26B-A4B against 31B dense (model card and tech report, arXiv 2607.02770):

Read the shape, not the average. On short single-turn benchmarks the MoE gives up 0.4-2.6 points, which is nearly free, and 2.9 on LiveCodeBench v6. On long-context retrieval, multi-step reasoning and agentic work the gap opens to 9-22 points, and Codeforces Elo falls 432.

Qwen’s own Qwen3.5-27B model card shows the same pattern at 9x fewer active parameters: MMLU-Pro 85.3 vs 86.1 and GPQA-D 84.2 vs 85.5 are near-parity, while SWE-bench Verified is 69.2 vs 72.4, LiveCodeBench v6 74.6 vs 80.7, AA-LCR 58.5 vs 66.1 and VITA-Bench 31.9 vs 41.9. The MoE does win CodeForces (2028 vs 1899) and TAU2 (81.2 vs 79.0), so this is not uniform.

There is a mechanism for the pattern. arXiv 2604.24827 finds MoE factual knowledge tracks total parameters (R^2=0.67) considerably better than active (R^2=0.41). Knowledge scales with what you store; reasoning depth scales with what you compute.

One myth to drop: the “a MoE is equivalent to a dense model of sqrt(total x active)” rule has no primary source. It is absent from the scaling-law literature. Epoch AI’s MoE-vs-dense inference analysis uses fractional exponents, total / (E^0.44 / k^0.63), not a square root.

What the benchmarks miss in daily use

Benchmarks score answers at fixed context lengths. Three costs decide whether a model feels usable on a Mac, and we have measured them for the dense Gemma 4 31B on our test machine; we have not measured the 26B MoE the same way. The first is prompt processing. The 31B reads prompts at about 65 tokens a second, so a 5,197-token prompt waited 79.56 seconds before the first token, roughly 75 percent of the wait for a 290-token answer, in our prompt processing benchmark. A retrieval workflow, where the question plus retrieved context can reach 4,000 tokens, spends most of its wait there.

The second is thinking mode. In the same runs, 254 to 297 thinking tokens came before 35 to 36 visible ones, so roughly 87 percent of generation time went to text the user never reads. Leave thinking off by default and turn it on for hard problems. The third is agent loops: the 31B generated at 10.3 to 11.5 tokens a second, so a reply capped at 300 tokens took 19 to 28 seconds, and an agent that makes five tool calls pays that five times. That compound wait is where a MoE’s decode speed counts most, and it sits against the larger quality gap on agentic benchmarks in the table above. We weigh the same trade in our pick for the best local model for OpenCode.

How much RAM should you buy for a 26B MoE or a dense model on a Mac?

The figures in this section are our own, from later tests on our test machine, an M4 Pro with 48 GB of unified memory. We have not measured Gemma 4 26B-A4B itself. The nearest pair we have measured is a MoE and a dense model of similar size: Qwen3.6 35B-A3B and Gemma 4 31B.

Source: StrideNote hands-on testing on a 48 GB M4 Pro, September and October 2026, from Qwen3.6 35B-A3B vs Gemma 4 31B on a Mac, MLX vs llama.cpp in LM Studio and Gemma 4 31B MLX vs GGUF.

The MoE file was 3.22 GB larger than the dense one (22.07 minus 18.85) and generated about 4.7 times as fast on llama.cpp (50.8 divided by 10.9). It also finished far fewer of our reasoning runs, which matches the vendor tables: the speed is real and the cost shows up on long, multi-step work.

For Gemma 4 26B-A4B itself we have only the download. ollama pull gemma4:26b fetches 16 GB of weights, and the Q4 builds run about 16 to 18 GB depending on the variant; Ollama’s library lists 18 GB for Q4_K_M. On a 24 GB machine Q4_K_M is the realistic choice and leaves about 6 to 8 GB for the operating system, the KV cache and other applications, so keep context at 4K to 8K unless every other application is closed. With 32 to 48 GB, Q5_K_M is a reasonable upgrade if you want extra margin for long-context reasoning. Our guide to picking a local model for your RAM covers the other tiers.

For a purchase, budget the file plus context, then add room for the operating system. On Gemma 4 31B we measured 21.97 GB for the whole process at an 8K context and 26.33 GB at 64K once the context was in use (the same test also recorded 25.24 GB at a 65,536-token setting, 6.39 GB over the 18.85 GB file), in our context window RAM measurements. On those numbers a 16 GB Mac holds neither model, since both files are larger than the memory. A 32 GB Mac would hold the dense model at 8K with about 10 GB to spare (32 minus 21.97), and the MoE file with under 10 GB left (32 minus 22.07) before its own context, which we have not measured. A 48 GB Mac ran both, one at a time, on our test machine. We have not tested either model on a 32 GB machine.

On our measurements, the MoE of this size gave about 4.7 times the output of the dense model on llama.cpp, and the dense model finished more of our reasoning runs with a slightly smaller file. On LM Studio’s MLX engine the Qwen MoE also held about 64.7 tokens a second for each of two callers at once, in our parallel requests test. We have not measured Gemma 4 26B-A4B itself, the MoE’s context cost per token, or either model on a real 32 GB Mac, so the 32 GB figures above stay arithmetic. Those three measurements are the test that would settle the purchase, and they are on our list.

What is the short version on MoE against dense?

• Total parameters decide whether it loads; active parameters decide how fast it decodes. “3B active” is not a 3B memory footprint.
• One same-machine record on Apple Silicon, Ollama issue #15368, puts MoE at about 5x the decode speed of a similarly sized dense model.
• The MoE can be larger in RAM than a dense model with far more active parameters: 24 GB against 17 GB for the Qwen3.5 pair in Ollama’s library.
• MoE does not shrink the KV cache. At 40K+ context the cache, not the weights, dominates.
• Decode is bandwidth-bound; the expert-gather path is not even invoked below batch 32. Gather optimisations are prefill wins (6.42x prefill, 1.03x decode).
• MTP speculative decoding loses on Metal in llama.cpp issue #23752, another sign that decode is bandwidth-bound.
• Quality cost is small on short tasks (0.4-2.6 points) and large on long-context and agentic work (9-22 points, -432 Codeforces Elo).
• The “sqrt(total x active)” equivalence rule has no source. Do not use it.

Where the record is silent

Worth naming, because these absences shape how much confidence anyone should have:

• No vendor has published a compute-matched MoE-vs-dense ablation for any shipped model. Google’s Gemma 4 tech report contains no MoE ablation and no stated rationale for building the MoE variant. Qwen ships side-by-side tables but no controlled comparison.
• No Apple or MLX maintainer statement on the MoE decode bottleneck. The bandwidth conclusion has to be read off dispatch thresholds in the source, not from an official answer.
• No published quality comparison of MoE vs dense measured on a Mac. The quality tables are vendor datacenter evals; the Mac comparisons are throughput-only. Nobody has joined the two.
• SimpleQA appears in none of the Gemma 4, Qwen3.5 or Llama 4 cards, which, given that knowledge tracks total parameters, is the single most informative missing benchmark.
• No expert-cache hit-rate study across models. The llama.cpp offloading PR reports more than 90% for one model at one cache size, but nobody has related hit rate to expert count, routing entropy or context length.

Also live as of July 2026: open correctness bugs in MLX’s sorted gather_qmm path on M5/NAX hardware (issues #3856 and #3887) causing silent numerical corruption at certain shapes. Prefill-path, but worth knowing before you trust a long-prompt result on a new Mac.

Experts can also be removed outright rather than compressed. In REAP-pruned MoE builds, covered in our GGUF build audit, a quarter of the experts are deleted before quantisation, and the count in the metadata changes to match.

Disclaimer: The architecture, speed and quality comparisons in this piece come from project records and vendor documentation, each named. The RAM section and the daily-use figures for Gemma 4 31B and Qwen3.6 35B-A3B are our own measurements on our test machine. We have not run Gemma 4 26B-A4B ourselves; where the published record is silent we say so.