Speculative decoding and MTP are one trick in two places

A local model writes one token at a time, and on our test machine a dense 27B manages about nine of them a second. Speculative decoding and multi-token prediction both attack that limit the same way: guess several tokens ahead cheaply, then let the big model check the guesses in one pass. The difference in speculative decoding vs MTP is where the guesses come from, and on a Mac that difference decides whether the speed-up is in the file you downloaded at all.

This piece sets the two side by side, shows how llama.cpp and Ollama switch each one on, reports the acceptance rates we measured for two Qwen models on a 48 GB M4 Pro, turns those speeds into replies an hour for a team sharing one server, and shows what we found inside the GGUF files that decide whether MTP is available at all.

What is the difference between speculative decoding and MTP?

Speculative decoding came first. The 2022 paper Fast Inference from Transformers via Speculative Decoding describes a small draft model that proposes tokens and a large target model that verifies them together. Tokens the target model agrees with are kept, and the first disagreement is replaced with the target model’s own choice. The paper states the outputs are identical to normal decoding, and it reports a 2X to 3X acceleration on T5-XXL. That figure is the paper’s own claim, on its own hardware.

The catch is the draft model. It has to share the target’s vocabulary and be much faster, which in practice means a separate, smaller network you download and load beside the main one.

Multi-token prediction moves the drafting inside the model. Extra layers are trained alongside the main network to predict tokens beyond the next one, so the model carries its own draft head. DeepSeek-AI’s DeepSeek-V3 Technical Report, published on 27 December 2024, describes a model that “sets a multi-token prediction training objective for stronger performance”. A runtime can then use those extra layers as the draft step for speculative decoding.

So MTP is not a rival to speculative decoding. It is one way to supply the draft. The checking step is the same in both cases, which is why neither should change what the model writes.

Which terms do the primary sources use?

“MTP drafter” is a phrase people search for, but it is not a term in any spec or codebase we checked. The terms that carry weight in primary sources are draft model (llama.cpp’s --model-draft, Ollama’s DRAFT) and proposer, which is what vLLM’s code calls these components: its vllm/v1/spec_decode/ directory has ngram_proposer.py, and eagle.py contains zero occurrences of “drafter”. “Drafter” lives in blog prose. “MTP draft head” or “MTP module” is the safe phrasing.

One error not to repeat: Ollama’s v0.23.1 release note heading expands MTP as “Multi-token Processing”. That is a typo on Ollama’s side. The pull request, the parameter docs and Ollama’s June blog all say prediction.

How do llama.cpp and Ollama switch on MTP?

llama.cpp’s server documentation lists the speculative decoding types in one option, --spec-type, with “default: none”. The accepted values include draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark and several ngram variants. The server documentation gives the list without describing each type. llama.cpp’s docs/speculative.md does describe one: draft-mtp is documented there as “Use Multi Token Prediction (MTP) heads from the main model.”

Two more options set how far ahead to guess. --spec-draft-n-max is the “number of tokens to draft for speculative decoding (default: 3)”, and --spec-draft-n-min is the minimum, with a default of 0. The documentation notes that the older --draft and --draft-max arguments have been removed in favour of these. A separate draft model is passed with --model-draft, described as the “draft model for speculative decoding”.

Hermes Agent’s built-in engine, which runs llama.cpp build b10679, writes these settings for the Qwen models in its catalog. For Qwen3.8 27B its generated launch policy was spec-type = draft-mtp with spec-draft-n-max = 2, plus backend sampling for both the target and the draft. We walked through that engine in our guide to a private Hermes Agent assistant on a Mac. When Qwen3.6 35B-A3B loaded, the server log recorded “creating MTP draft context against the target model” nine seconds into the load.

Other servers take their own flags. The Qwen3.6-27B model card lists “MTP: trained with multi-steps” and gives vLLM’s --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' and SGLang’s --speculative-algo NEXTN.

Ollama: on by default for Gemma 4 on Macs

Ollama reached the same idea by a different route. If you run Gemma 4 on Apple silicon through Ollama, you have been getting speculative decoding since 30 June 2026, whether you configured it or not. Ollama’s own figure is nearly 90% faster token generation across a coding-agent benchmark, on by default, with no flags.

Ollama PR #15980, merged on 5 May, is the one to read. It added the MTP path, the DRAFT Modelfile instruction, safetensors draft-model import, and --quantize-draft. The v0.31.1 note is the one that matters for daily use: “Ollama auto-tunes how many tokens to draft as it runs, so the speedup is on by default, requires no configuration, and does not change the model’s output.”

Part of the gain came from Ollama contributing a batched-draft-token matmul kernel upstream to MLX. On an M5 Max with NVFP4 weights, that makes Gemma 4’s largest matmuls 2 to 2.5x faster. It is the same MLX-only pattern as NVFP4 on Apple Silicon: the local-inference work is landing in the Apple path first.

DRAFT is a valid Modelfile instruction. We verified it in parser/parser.go, where it sits in the valid-command list alongside from, adapter and template. It is not in the public Modelfile reference. That is the same shape as the OpenJarvis analytics switch we found in our retest: a working control the documentation does not mention.

What is documented is the parameter, and it carries a trap. The docs describe draft_num_predict as the “Maximum number of speculative draft tokens to predict per step when a draft model is available. Separate draft models default to 4; embedded MTP tensors require setting this parameter. Set to 0 to disable speculative drafting.” Since v0.31.1 auto-tunes the draft count, the second clause sits in tension with the release note, and we could not resolve from the docs alone which wins on a current build. To turn drafting off entirely, set draft_num_predict: 0.

Qwen is the frustrating case. Ollama issue #17019, opened on 3 July 2026, is about Qwen3.5 and 3.6 safetensors whose mtp.* tensors were corrupting MLX imports. The fix dropped them. So Qwen ships the heads and Ollama currently discards them. Lower confidence, from survey material rather than individual model cards: GLM-5, Qwen3-Next-80B-A3B, Nemotron 3 Super, MiniMax M2.7 and MiMo are also reported to ship MTP heads. Treat those as leads, not facts.

There is still no general speculative decoding on Ollama’s llama.cpp engine. Ollama issue #5800, “Enable speculative decoding”, has been open since 19 July 2024 with 64 comments, and the community pull request #8134 was closed unmerged in January 2025. Ollama bundles llama.cpp and exposes none of its draft flags. MTP on Ollama is a macOS, MLX, Gemma 4 feature; on GGUF, run llama.cpp directly.

For a team whose developers run Gemma 4 on Apple silicon, the change arrives with a software update. We have not measured it ourselves, so treat the 90% as Ollama’s number. A team that standardised on Qwen gets nothing from it today, and a team serving from Linux machines through Ollama’s GGUF path gets no speculative decoding at all. Any internal benchmark of Gemma 4 on Ollama now measures MTP as well as the model, so set draft_num_predict: 0 when the aim is to compare models or formats, and record the Ollama version with every result. Because the draft count auto-tunes, a speed quoted from one machine may not carry to another.

These are the checks we intend to run on our M4 Pro, in order. None needs more than a Mac with Gemma 4 pulled.

1. Compare tokens a second on Gemma 4 with draft_num_predict unset against draft_num_predict: 0. If there is no difference, MTP is not engaging and the auto-tune claim needs qualifying.
2. Find out whether an explicit draft_num_predict value beats auto-tune on a current build.
3. Run the same seed and prompt with MTP on and off. The claim is that output is identical, and it is cheap to check.
4. Test it under an agent, not a chat. Speculative decoding wins most on predictable token runs, and agent traffic is full of them. The tool-calling test from our tool-calling piece is the right load.
5. Check acceptance rate, not just tokens a second. A high draft count with poor acceptance can be slower than no drafting.

What MTP acceptance rates did we measure on an M4 Pro?

On 11 September we ran both Qwen models through Hermes’s llama.cpp engine with MTP on, during the ARC-AGI-3 board test in Qwen3.6 35B-A3B vs Gemma 4 31B on a Mac. The server prints a draft acceptance line for every request: the share of drafted tokens the target model kept, and the mean number accepted per step.

Two things stand out. Acceptance on Qwen3.6 was high enough that each verification step kept close to three tokens on average. Qwen3.8 accepted less, and even with MTP on it generated fewer tokens a second than Gemma 4 31B did without any draft.

We did not run either Qwen model with MTP switched off on this machine, so we cannot say how much faster MTP made them. A speed-up figure needs that second run, and we will not estimate one.

High acceptance also needs care in reading. A draft is easy to accept when the next tokens are predictable, and text that repeats itself is predictable. The 96.8% Qwen3.6 run came from a test in which we later found one capped run looping, with 99.5% of the last quarter of its reasoning repeating earlier text. Acceptance measures how often the guesses matched, not whether the text was worth writing.

How many replies an hour does speculative decoding vs MTP give a shared server?

The measured generation speeds are the starting point. With MTP on, Qwen3.6 35B-A3B ran at a median 50.8 tokens a second across 12 runs in our Qwen3.6 and Gemma 4 comparison, Qwen3.8 27B at 8.7, and Gemma 4 31B, with no draft at all, at 10.9. An hour is 3,600 seconds, so those are about 182,900, 31,300 and 39,200 generated tokens an hour for one caller, before prompt processing, which adds its own time to every request. On a reasoning model the thinking counts towards the reply length too.

The hour is shared, not multiplied. When two callers used one LM Studio endpoint, the server produced 11.2 tokens a second in total against 11.3 for one caller, so each got half. We have not repeated that test on Hermes’s llama.cpp engine.

These figures do not say how many replies MTP added, and they do not say how many replies an hour a shared server gives. What would settle both is the same Qwen3.6 prompts run with --spec-type none, and several concurrent callers against Hermes’s llama.cpp server with throughput measured under load, and both are on our list.

Does your GGUF file include the MTP head?

MTP only works if the draft layers survive into the file you run, and they do not always. The draft ships in two different shapes depending on the model family, and in neither case does every download contain it. The same component has three fates: the model author decides whether to train an MTP component and how to package it, the quantiser decides whether to carry it across, and the runtime decides whether to use what it finds. A speed-up that is real at every one of those layers still reaches you only if all three line up.

Qwen3.5: one build keeps the head, another drops it

We read the tensor tables of seven Qwen3.5-35B-A3B builds from three publishers, about 16 MB per file. unsloth’s builds contain blocks 0 through 39. bartowski’s contain blocks 0 through 40. mradermacher’s, like unsloth’s, stop at 39. Searching for tensors whose names mark them as multi-token prediction components returns four in bartowski’s build and zero in unsloth’s. Block 40 is where they live. A build without that block cannot use draft-mtp, whatever the flag says.

The keeping was deliberate. In bartowski’s builds, block 40 is not quantised like its neighbours. Across the expert tensor classes the file runs at IQ3_S or IQ2_XXS, and block 40 sits at Q8_0, the highest precision in the file.

We also checked for silently dropped experts, a claim that circulates whenever a large mixture-of-experts model gets community quants. Qwen3.5-35B-A3B declares 256 experts with 8 active per token, in an architecture the metadata calls qwen35moe. In all seven builds, the expert dimension of every fused expert tensor matches the declared expert count. When we audited eight builds of Gemma 4 26B-A4B we found no evidence of it either, so this is the second model family where it does not appear. Gemma 4 fuses its gate and up projections into a single ffn_gate_up_exps tensor, giving two expert tensor classes per layer. Qwen3.5 keeps them separate, giving three: ffn_gate_exps, ffn_up_exps and ffn_down_exps. A script that assumes two classes will silently miss a third of the expert weights.

Each quantiser protects different layers:

• unsloth’s MXFP4 build promotes exactly one block out of forty, block 10, in all three expert classes. A single interior block, which matches no obvious rule.
• bartowski’s IQ2_XXS promotes the first five blocks and several near the end on ffn_down_exps, plus block 40 across the board.
• bartowski’s IQ3_M splits ffn_down_exps almost in half: twenty layers at Q5_K, twenty at IQ3_S, on a schedule that skips blocks 7 and 10.
• mradermacher’s Q4_K_M uses that same alternating shape, twenty Q6_K against twenty Q4_K, on very similar indices.

The count of distinct quantisation types in a single file ranges from three to six across this set, the same spread we measured on Gemma. The quant name is not a reliable guide either: unsloth’s Q3_K_M build for this model contains no Q3_K tensors at all; its expert weights are IQ3_XXS and IQ4_XS.

Hermes’s catalog downloads Qwen3.6 from an unsloth repository with MTP in its name, unsloth/Qwen3.6-35B-A3B-MTP-GGUF, which is the only sign in a listing that the draft layers are there. The same split shows up in Nemotron, whose 4-bit GGUF we examined for its MTP layers.

Gemma 4 12B: the draft is a separate model

Gemma 4 packages its draft the other way. We read six Gemma 4 12B files across three publishers: unsloth, Google’s own quantisation-aware-trained release, and bartowski. All three produce a structurally identical main model, 48 blocks and 667 tensors, differing only in quantisation mix: unsloth ships Q4_K with Q6_K promotions, Google ships Q4_0 with Q6_K promotions, and bartowski ships Q4_K, Q5_K and Q6_K. Not one embeds an MTP head in the main file. That includes Google’s own quantisation-aware-trained release, the build most likely to be treated as the reference version.

Only bartowski’s repository publishes the draft, as mtp-gemma-4-12B-it-Q4_0.gguf alongside the ordinary quants. Reading it returns a small complete model, not a fragment:

Two of those 49 tensors are nextn.pre_projection and nextn.post_projection. The draft model projects into and out of the main model’s representation space so its proposals can be checked cheaply. Four blocks against the main model’s forty-eight is roughly a twelfth of the network, carrying the full vocabulary so it can propose real tokens.

The two shapes follow from training. Qwen trains its MTP tensors as part of the network, so they live in the same tensor namespace and a converter either carries them across or drops them. Google trains a separate assistant model, so it converts into its own file with its own architecture string. gemma4-assistant is an architecture a runtime must recognise; a loader that knows gemma4 will not necessarily know what to do with it. Ollama’s notes treat Gemma 4’s draft as MTP tensors it uses without a second download, while the 12B GGUF files we read keep the draft outside the main file, so the packaging may differ between Ollama’s own Gemma 4 tags and the GGUF builds.

We have not run Gemma 4 12B with and without this draft file on our test machine, so we give no speed-up, no acceptance rate and no figure for the memory the draft adds. That run, with bartowski’s draft beside the main model, is on our list. We have also not run bartowski’s Qwen3.5-35B-A3B build with --spec-type draft-mtp against --spec-type none; that is on the list too.

What should a team check before relying on MTP?

The build you standardise on fixes which speed options exist for the life of the deployment. Each of these checks costs no more than a file listing or a range request over the first few megabytes of the file, the same method that shows what a quant name does and does not tell you.

Look for a separate file with an mtp- prefix. If the family packages its draft separately, the main quant will never contain it whichever publisher you choose. If the family embeds the head instead, compare block counts between publishers and against the layer count on the publisher’s own model card: a build one block shorter than another build of the same model has dropped the head, and a GGUF with one more block than the card is carrying something extra rather than something wrong. Nothing in the quant label records either.

Check the architecture string, not the filename, because that string decides whether your runtime can load the file. Do not read a small draft file as an aggressively compressed quant. It is a different network with its own job, and loading it as your primary model would give you a four-block model wearing a familiar name. And count the expert tensor classes, two on Gemma and three on Qwen, if you are auditing how a mixture-of-experts model sizes in memory, since missing a class understates the expert budget substantially.

Write down the publisher, the quant level and the block count of each file you approve, with the Hugging Face revision you pulled, so a later download cannot swap a 41-block file for a 40-block one under the same model name. For Gemma, the approval record has to name two files, two publishers and two revisions, and an update to one does not update the other; the check before rollout is whether your server loads both files and reports drafted tokens accepted. If your runtime is Ollama, which discards Qwen’s heads, the 40-block Qwen builds lose no option Ollama would have used. If you plan capacity around MTP on llama.cpp, start from a build that has the 41st block and measure with the draft on and off before you size hardware.

None of this makes any of these publishers wrong. Dropping an MTP head is defensible if your target runtime discards it anyway, and keeping it is defensible if you would rather ship what the model came with. Both choices arrive under the same model name and quant labels, and the measurement still missing on our machine is the plain one: the same Qwen3.6 prompt with --spec-type none, which would turn the acceptance rates above into a speed-up.