MoE on a Mac: 4 to 5 times faster, and bigger in memory than the dense model it beats

The short finding: a same-machine comparison recorded in Ollama’s issue tracker puts a mixture-of-experts model at about five times the decode speed of a dense model of similar size on Apple Silicon. And Ollama’s own model library shows a MoE that needs more memory than a dense model with nine times its active parameters. Both facts follow from the same rule, and the rule is the thing to learn.

The rule: total loads, active computes

Google states the convention plainly on the Gemma 4 model card: “the ‘A’ in 26B A4B stands for active parameters.”

Every expert must be resident, because the router can select any of them at any token. Only the routed ones do arithmetic. A model labelled “3B active” therefore still needs memory for every parameter it has, 35 billion in the case of Qwen3.5-35B-A3B.

The rule holds at the largest scale. Qwen3.8-Max, whose weights Alibaba published in August, activates 95 billion of its 2.4 trillion parameters per token, and all 2.4 trillion have to be held in memory.

The architectures now shipping locally:

Llama 4 uses a different convention (17Bx16E means 17B active across 16 experts) and Meta routes each token to “the shared expert and also to one of the 128 routed experts”, i.e. top-1, unusually sparse.

A same-machine comparison

One comparison of the two designs on the same Mac appears in an official project record. Ollama issue #15368, filed in April against Ollama 0.20.2 on an M5 Max with 128 GB of unified memory, reports that Ollama’s llama.cpp runner reached about 75 tok/s on Gemma 4 26B-A4B and about 15 tok/s on the 31B dense model.

That is roughly a fivefold gap on an eightfold cut in active parameters (3.8B against 30.7B), which is about what the bandwidth argument below predicts. Treat it as one data point: the figures are rounded, and the report was written while debugging a flash-attention hang, not as a benchmark.

Why: bandwidth, and only bandwidth

Apple Silicon is bandwidth-bound on dense decode. Apple’s specifications put the M4 Pro at 273 GB/s of memory bandwidth, half the M4 Max’s 546 GB/s, and on StrideNote’s M4 Pro a dense Gemma 4 31B generates a median of 11.0 tokens a second. The rest of our measurements on this machine are in our M4 Pro benchmarks.

MoE attacks exactly that. Fewer weights streamed per token means less to read, and reading is the constraint.

We expected a second effect and were wrong about it. Our working assumption was that immature expert-routing kernels on Metal and MLX would eat part of the theoretical win. The source code says otherwise, and the correction is more interesting than the guess.

In llama.cpp’s ggml-metal-ops.cpp:

“cpp

// ne21 = n_rows (batch size)

const int ne21_mm_id_min = 32;

if (props_dev->has_simdgroup_mm && ne00 >= 64 && (ne21 >= ne21_mm_id_min)) {

// kernel_mul_mm_id_map0 (expert id-map gather) + kernel_mul_mm_id

} else {

// kernel_mul_mv_id

}

“

Below batch 32, which is all decode, Metal never runs the expert gather path at all. It uses a plain per-expert matrix-vector kernel. MLX has the same shape of threshold: GatherQMM::eval_gpu requires M == 1 && B >= 16 && right_sorted_ && B / E >= 4, conditions a single decode token cannot meet for top-8-of-128, and mlx_lm/switch_layers.py sorts only when indices.size >= 64.

So expert-gather cost is structurally a prefill concern. Every published gather optimisation confirms it. llama.cpp PR #12612 is the cleanest datapoint: DeepSeek V2 Lite prefill went 150.94 to 968.34 t/s, a 6.42x gain, while decode moved 94.87 to 97.81 t/s, 1.03x. PRs #13388 and #15541 are the same story at 2.21x and 1.58x prefill.

MoE decode on Metal is memory-bandwidth-bound, and there is corroborating evidence in a place you would not look for it: MTP speculative decoding loses on Metal. llama.cpp issue #23752 reports Qwen3.5-9B-Q4_K_M dropping from 25.3 tok/s baseline to 19.3 at n_max 6, because the extra weight reads cost more than the saved steps. That is the signature of a bandwidth-bound system.

It also suggests a limit worth testing before you rely on it: if you already run MoE, MTP may have less left to give you, because two optimisations chasing the same bottleneck do not stack. No official project record we could find measures MTP on a MoE and a dense model side by side.

One source to treat carefully: arXiv 2604.18788 (NPUMoE) does say “expert routing is unpredictable and introduces dynamic tensor shapes”, claiming 1.32-5.55x latency gains. But that is the Apple Neural Engine, not the Metal GPU, a different machine with static-shape constraints the GPU does not have. It is not evidence about MLX or llama.cpp.

The memory inversion

Here is where the two-number rule bites.

These are download sizes from Ollama’s official model library (qwen3.5, gemma4), the weights alone before any KV cache.

The Qwen MoE is larger than the dense model beside it, even though it computes with a ninth of the active parameters. There is no contradiction: footprint tracks total parameters, so a 35B MoE outweighs a 27B dense, and a 25.2B MoE undercuts a 30.7B dense. Speed tracks active parameters regardless.

The practical rule: size your RAM against the total, and your speed expectations against the active count. A “3B active” model is not a 3B model on your memory budget.

MoE does not shrink the KV cache, and this is the part that decides long-context work. Attention layers are untouched by expert sparsity, so expert count is irrelevant here. What matters is the attention configuration, and it varies sharply. Qwen3.5-35B-A3B runs 3 GatedDeltaNet layers to 1 full attention, replacing a growing KV cache with fixed-size recurrent state in 75% of layers. Gemma 4 26B-A4B runs 25 sliding-window to 5 full attention across 40 layers.

At long context the cache dominates the weights outright. SharpAI’s SwiftLM project reports Gemma 4 26B on an M5 Pro at 40K context needing 54.8 GB with an FP16 KV cache, falling to 23.9 GB with quantized KV; at 100K, 54.3 GB falling to 26.4 GB. That matches what we saw ourselves with dense models in our tool-calling work, where Gemma 4 31B at 64K context took about 35 GB and generated at roughly 2 tok/s.

What the speed costs you in quality

Both vendors publish MoE against dense in the same table, which is unusually honest and worth using.

Gemma 4 26B-A4B against 31B dense (model card and tech report, arXiv 2607.02770):

Read the shape, not the average. On short single-turn benchmarks the MoE gives up 0.4-2.6 points, which is nearly free. On long-context retrieval, multi-step reasoning and agentic work the gap opens to 9-22 points, and Codeforces Elo falls 432.

Qwen’s own Qwen3.5-27B model card shows the same pattern at 9x fewer active parameters: MMLU-Pro 85.3 vs 86.1 and GPQA-D 84.2 vs 85.5 are near-parity, while SWE-bench Verified is 69.2 vs 72.4, LiveCodeBench v6 74.6 vs 80.7, AA-LCR 58.5 vs 66.1 and VITA-Bench 31.9 vs 41.9. The MoE does win CodeForces (2028 vs 1899) and TAU2 (81.2 vs 79.0), so this is not uniform.

There is a mechanism for the pattern. arXiv 2604.24827 finds MoE factual knowledge tracks total parameters (R^2=0.67) considerably better than active (R^2=0.41). Knowledge scales with what you store; reasoning depth scales with what you compute.

One myth to drop: the “a MoE is equivalent to a dense model of sqrt(total x active)” rule has no primary source. It is absent from the scaling-law literature. Epoch AI’s MoE-vs-dense inference analysis uses fractional exponents, total / (E^0.44 / k^0.63), not a square root.

If it does not fit: expert offloading

Because only a few experts fire per token, you can keep most of them off the GPU. Nothing is merged in either project, but the published results are striking.

The clearest record is llama.cpp PR #23440 (open, not merged), on Qwen3-30B-A3B-Q6_K, M3 Pro 36GB: vanilla 38.1 tok/s; 80 slots 29.1 tok/s at ~18.9 GiB wired (>90% hit rate, ~8.3 GiB saved); 16 slots 15.7 tok/s at ~7.8 GiB. Also 13 tok/s on an M1 Pro 16GB.

Note what that trade actually is: residency buys capacity, not speed. mlx-lm PR #1588 (closed without merging) states the cost plainly: when the model already fits, offloading is “5x slower decode and 8 to 12x slower prefill.”

Where the record is silent

Worth naming, because these absences shape how much confidence anyone should have:

• No vendor has published a compute-matched MoE-vs-dense ablation for any shipped model. Google’s Gemma 4 tech report contains no MoE ablation and no stated rationale for building the MoE variant. Qwen ships side-by-side tables but no controlled comparison.
• No Apple or MLX maintainer statement on the MoE decode bottleneck. The bandwidth conclusion has to be read off dispatch thresholds in the source, not from an official answer.
• No published quality comparison of MoE vs dense measured on a Mac. The quality tables are vendor datacenter evals; the Mac comparisons are throughput-only. Nobody has joined the two.
• SimpleQA appears in none of the Gemma 4, Qwen3.5 or Llama 4 cards, which, given that knowledge tracks total parameters, is the single most informative missing benchmark.
• No expert-cache hit-rate study across models. The llama.cpp offloading PR reports more than 90% for one model at one cache size, but nobody has related hit rate to expert count, routing entropy or context length.

Also live as of July 2026: open correctness bugs in MLX’s sorted gather_qmm path on M5/NAX hardware (issues #3856 and #3887) causing silent numerical corruption at certain shapes. Prefill-path, but worth knowing before you trust a long-prompt result on a new Mac.

The short version

• Total parameters decide whether it loads; active parameters decide how fast it decodes. “3B active” is not a 3B memory footprint.
• One same-machine record on Apple Silicon, Ollama issue #15368, puts MoE at about 5x the decode speed of a similarly sized dense model.
• The MoE can be larger in RAM than a dense model with far more active parameters: 24 GB against 17 GB for the Qwen3.5 pair in Ollama’s library.
• MoE does not shrink the KV cache. At 40K+ context the cache, not the weights, dominates.
• Decode is bandwidth-bound; the expert-gather path is not even invoked below batch 32. Gather optimisations are prefill wins (6.42x prefill, 1.03x decode).
• MTP speculative decoding loses on Metal in llama.cpp issue #23752, another sign that decode is bandwidth-bound.
• Quality cost is small on short tasks (0.4-2.6 points) and large on long-context and agentic work (9-22 points, -432 Codeforces Elo).
• The “sqrt(total x active)” equivalence rule has no source. Do not use it.

Experts can also be removed outright rather than compressed. In REAP-pruned MoE builds a quarter of the experts are deleted before quantisation, and the count in the metadata changes to match.

Disclaimer: This is a research piece. The measurements in this piece come from project records and vendor documentation, each named. We have not run these comparisons on our own hardware; where the published record is silent we say so.