MoE on a Mac: 4 to 5 times faster, and bigger in memory than the dense model it beats

This is a research piece. The measurements below are other people’s, attributed. We have not run these comparisons on our own hardware; where the published record is silent we say so.

The short finding: four independent published comparisons on Apple Silicon put a mixture-of-experts model at 4.2 to 5.2 times the decode speed of a dense model of similar size. And in two of them the MoE occupies more unified memory than the dense model it outruns. Both facts follow from the same rule, and the rule is the thing to learn.

The rule: total loads, active computes

Google states the convention plainly on the Gemma 4 model card: “the ‘A’ in 26B A4B stands for active parameters.”

Every expert must be resident, because the router can select any of them at any token. Only the routed ones do arithmetic. marcofariasmx puts the consequence better than we could: “‘3B active’ refers to compute, not footprint. All 35B parameters must reside in memory.”

The architectures now shipping locally:

Llama 4 uses a different convention (17Bx16E means 17B active across 16 experts) and Meta routes each token to “the shared expert and also to one of the 128 routed experts”, i.e. top-1, unusually sparse.

Four same-machine comparisons

This is the comparison that answers the question, and contrary to first impressions it has been published repeatedly.

Sources in order: Incept5/gemma4-benchmark (MLX 0.31.1, raw JSON and scripts published), Ollama issue #15368, stared/benching-local-llms-on-apple-silicon (14 June 2026, median of 3), and marcofariasmx/local-llm-apple-silicon.

The consistency across four setups, two model families, three quantization schemes and two chip generations is what makes this credible. A ratio of 4-5x on a 7-9x active-parameter reduction is roughly what the bandwidth argument predicts.

Two caveats. The Incept5 repo mislabels Gemma 4 E2B and E4B as MoE (they are MatFormer dense); the 26B-vs-31B row is unaffected but the methodology has a known error. And the Qwen head-to-heads use Qwen3.6, not Qwen3.5. Both series exist and should not be conflated.

Why: bandwidth, and only bandwidth

Apple Silicon is bandwidth-bound on dense decode. Our own M4 Pro benchmarks show it: 273 GB/s on the M4 Pro against 546 GB/s on the M4 Max, and a dense Gemma 4 31B at 8-12 tok/s against 15-25. Same parameters, double the bandwidth, most of the speedup.

MoE attacks exactly that. Fewer weights streamed per token means less to read, and reading is the constraint.

We expected a second effect and were wrong about it. Our working assumption was that immature expert-routing kernels on Metal and MLX would eat part of the theoretical win. The source code says otherwise, and the correction is more interesting than the guess.

In llama.cpp’s ggml-metal-ops.cpp:

“cpp

// ne21 = n_rows (batch size)

const int ne21_mm_id_min = 32;

if (props_dev->has_simdgroup_mm && ne00 >= 64 && (ne21 >= ne21_mm_id_min)) {

// kernel_mul_mm_id_map0 (expert id-map gather) + kernel_mul_mm_id

} else {

// kernel_mul_mv_id

}

“

Below batch 32, which is all decode, Metal never runs the expert gather path at all. It uses a plain per-expert matrix-vector kernel. MLX has the same shape of threshold: GatherQMM::eval_gpu requires M == 1 && B >= 16 && right_sorted_ && B / E >= 4, conditions a single decode token cannot meet for top-8-of-128, and mlx_lm/switch_layers.py sorts only when indices.size >= 64.

So expert-gather cost is structurally a prefill concern. Every published gather optimisation confirms it. ggerganov’s llama.cpp PR #12612 is the cleanest datapoint: DeepSeek V2 Lite prefill went 150.94 to 968.34 t/s, a 6.42x gain, while decode moved 94.87 to 97.81 t/s, 1.03x. PRs #13388 and #15541 are the same story at 2.21x and 1.58x prefill.

MoE decode on Metal is memory-bandwidth-bound, and there is corroborating evidence in a place you would not look for it: MTP speculative decoding loses on Metal. llama.cpp issue #23752 reports Qwen3.5-9B-Q4_K_M dropping from 25.3 tok/s baseline to 19.3 at n_max 6, because the extra weight reads cost more than the saved steps. That is the signature of a bandwidth-bound system.

Which produces a genuinely useful, non-obvious result from stared’s data: MTP helps unevenly. +75% on the dense 27B (17 to 32 tok/s), only +12% on the MoE 35B-A3B. If you are already running MoE, MTP has much less left to give you. Two optimisations chasing the same bottleneck do not stack.

One source to treat carefully: arXiv 2604.18788 (NPUMoE) does say “expert routing is unpredictable and introduces dynamic tensor shapes”, claiming 1.32-5.55x latency gains. But that is the Apple Neural Engine, not the Metal GPU, a different machine with static-shape constraints the GPU does not have. It is not evidence about MLX or llama.cpp.

The memory inversion

Here is where the two-number rule bites.

The MoE is *larger* in RAM in two of three, while being 4.8-5.2x faster. There is no contradiction: footprint tracks total parameters, so a 35B MoE outweighs a 27B dense, and a 25.2B MoE undercuts a 30.7B dense. Speed tracks active parameters regardless.

The practical rule: size your RAM against the total, and your speed expectations against the active count. A “3B active” model is not a 3B model on your memory budget.

MoE does not shrink the KV cache, and this is the part that decides long-context work. Attention layers are untouched by expert sparsity, so expert count is irrelevant here. What matters is the attention configuration, and it varies sharply. Qwen3.5-35B-A3B runs 3 GatedDeltaNet layers to 1 full attention, replacing a growing KV cache with fixed-size recurrent state in 75% of layers. Gemma 4 26B-A4B runs 25 sliding-window to 5 full attention across 40 layers.

At long context the cache dominates the weights outright. SharpAI’s SwiftLM project reports Gemma 4 26B on an M5 Pro at 40K context needing 54.8 GB with an FP16 KV cache, falling to 23.9 GB with quantized KV; at 100K, 54.3 GB falling to 26.4 GB. That matches what we saw ourselves with dense models in our tool-calling work, where Gemma 4 31B at 64K context took about 35 GB and generated at roughly 2 tok/s.

What the speed costs you in quality

Both vendors publish MoE against dense in the same table, which is unusually honest and worth using.

Gemma 4 26B-A4B against 31B dense (model card and tech report, arXiv 2607.02770):

Read the shape, not the average. On short single-turn benchmarks the MoE gives up 0.4-2.6 points, which is nearly free. On long-context retrieval, multi-step reasoning and agentic work the gap opens to 9-22 points, and Codeforces Elo falls 432.

Qwen’s own Qwen3.5-27B model card shows the same pattern at 9x fewer active parameters: MMLU-Pro 85.3 vs 86.1 and GPQA-D 84.2 vs 85.5 are near-parity, while SWE-bench Verified is 69.2 vs 72.4, LiveCodeBench v6 74.6 vs 80.7, AA-LCR 58.5 vs 66.1 and VITA-Bench 31.9 vs 41.9. The MoE does win CodeForces (2028 vs 1899) and TAU2 (81.2 vs 79.0), so this is not uniform.

There is a mechanism for the pattern. arXiv 2604.24827 finds MoE factual knowledge tracks total parameters (R^2=0.67) considerably better than active (R^2=0.41). Knowledge scales with what you store; reasoning depth scales with what you compute.

One myth to drop: the “a MoE is equivalent to a dense model of sqrt(total x active)” rule has no primary source. It is absent from the scaling-law literature. Epoch AI’s MoE-vs-dense inference analysis uses fractional exponents, total / (E^0.44 / k^0.63), not a square root.

If it does not fit: expert offloading

Because only a few experts fire per token, you can keep most of them off the GPU. Nothing is merged in either project, but the published results are striking.

The flash-moe project by danveloper runs Qwen3.5-397B-A17B on a 48 GB M3 Max at 4.36 tok/s, streaming 209 GB from SSD at a measured 17.5 GB/s with only ~6 GB resident. The finding we did not expect: “Trust the OS.” No custom cache. The OS page cache achieves roughly a 71% expert hit rate on plain LRU, and “every custom caching approach we tested was slower due to GPU memory pressure or overhead.”

The Metal-native counterpart is llama.cpp PR #23440 (draft), on Qwen3-30B-A3B-Q6_K, M3 Pro 36GB: vanilla 38.1 tok/s; 80 slots 29.1 tok/s at ~18.9 GiB wired (>90% hit rate, ~8.3 GiB saved); 16 slots 15.7 tok/s at ~7.8 GiB. Also 13 tok/s on an M1 Pro 16GB.

Note what that trade actually is: residency buys capacity, not speed. mlx-lm PR #1588 (open) states the cost plainly: when the model already fits, offloading is “5x slower decode and 8 to 12x slower prefill.”

Where the record is silent

Worth naming, because these absences shape how much confidence anyone should have:

• No vendor has published a compute-matched MoE-vs-dense ablation for any shipped model. Google’s Gemma 4 tech report contains no MoE ablation and no stated rationale for building the MoE variant. Qwen ships side-by-side tables but no controlled comparison.
• No Apple or MLX maintainer statement on the MoE decode bottleneck. MLX issue #3402 and discussion #3209 both went without maintainer response. The bandwidth conclusion has to be read off dispatch thresholds in the source, not from an official answer.
• No published quality comparison of MoE vs dense measured on a Mac. The quality tables are vendor datacenter evals; the Mac comparisons are throughput-only. Nobody has joined the two.
• SimpleQA appears in none of the Gemma 4, Qwen3.5 or Llama 4 cards, which, given that knowledge tracks total parameters, is the single most informative missing benchmark.
• No expert-cache hit-rate study across models. Three implementations converge near 70%, but nobody has related hit rate to expert count, routing entropy or context length.

Also live as of July 2026: open correctness bugs in MLX’s sorted gather_qmm path on M5/NAX hardware (issues #3856 and #3887) causing silent numerical corruption at certain shapes. Prefill-path, but worth knowing before you trust a long-prompt result on a new Mac.

The short version

• Total parameters decide whether it loads; active parameters decide how fast it decodes. “3B active” is not a 3B memory footprint.
• Four published Apple Silicon comparisons: MoE is 4.2 to 5.2x faster than a similarly sized dense model.
• The MoE can be larger in RAM than the dense model it beats, 22.4 GB against 17.9 GB in one measurement.
• MoE does not shrink the KV cache. At 40K+ context the cache, not the weights, dominates.
• Decode is bandwidth-bound; the expert-gather path is not even invoked below batch 32. Gather optimisations are prefill wins (6.42x prefill, 1.03x decode).
• Consequence: MTP adds little on top of MoE (+12% vs +75% on dense), both chase the same bottleneck.
• Quality cost is small on short tasks (0.4-2.6 points) and large on long-context and agentic work (9-22 points, -432 Codeforces Elo).
• The “sqrt(total x active)” equivalence rule has no source. Do not use it.