A local model writes one token at a time, and on our test machine a dense 27B manages about nine of them a second. Speculative decoding and multi-token prediction both attack that limit the same way: guess several tokens ahead cheaply, then let the big model check the guesses in one pass. The difference in speculative decoding vs MTP is where the guesses come from, and on a Mac that difference decides whether the speed-up is in the file you downloaded at all.
This piece sets the two side by side, shows how llama.cpp switches each one on, and reports the acceptance rates we measured for two Qwen models on a 48 GB M4 Pro.
What is the difference between speculative decoding and MTP?
Speculative decoding came first. The 2022 paper Fast Inference from Transformers via Speculative Decoding describes a small draft model that proposes tokens and a large target model that verifies them together. Tokens the target model agrees with are kept, and the first disagreement is replaced with the target model’s own choice. The paper states the outputs are identical to normal decoding, and it reports a 2X to 3X acceleration on T5-XXL. That figure is the paper’s own claim, on its own hardware.
The catch is the draft model. It has to share the target’s vocabulary and be much faster, which in practice means a separate, smaller network you download and load beside the main one.
Multi-token prediction moves the drafting inside the model. Extra layers are trained alongside the main network to predict tokens beyond the next one, so the model carries its own draft head. DeepSeek-AI’s DeepSeek-V3 Technical Report, published on 27 December 2024, describes a model that “sets a multi-token prediction training objective for stronger performance”. A runtime can then use those extra layers as the draft step for speculative decoding.
So MTP is not a rival to speculative decoding. It is one way to supply the draft. The checking step is the same in both cases, which is why neither should change what the model writes.
| Separate draft model | MTP head | |
|---|---|---|
| Where the draft comes from | A second, smaller model | Extra layers trained into the main model |
| Extra download | Yes | No, if the build kept the layers |
| Vocabulary match | Must be arranged | Shared by design |
| llama.cpp setting | --model-draft plus a draft type | --spec-type draft-mtp |
How llama.cpp turns on MTP: draft-mtp and the draft model flags
llama.cpp’s server documentation lists the speculative decoding types in one option, --spec-type, with “default: none”. The accepted values include draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark and several ngram variants. The documentation gives the list without describing each type.
Two more options set how far ahead to guess. --spec-draft-n-max is the “number of tokens to draft for speculative decoding (default: 3)”, and --spec-draft-n-min is the minimum, with a default of 0. The documentation notes that the older --draft and --draft-max arguments have been removed in favour of these. A separate draft model is passed with --model-draft, described as the “draft model for speculative decoding”.
Hermes Agent’s built-in engine, which runs llama.cpp build b10679, writes these settings for the Qwen models in its catalog. For Qwen3.8 27B its generated launch policy was spec-type = draft-mtp with spec-draft-n-max = 2, plus backend sampling for both the target and the draft. We walked through that engine in our guide to Hermes Agent local models on a Mac. When Qwen3.6 35B-A3B loaded, the server log recorded “creating MTP draft context against the target model” nine seconds into the load.
Ollama reached the same idea by a different route. It switched MTP on by default for Gemma 4 on Apple silicon, and its Modelfile parser accepts a DRAFT instruction the Modelfile reference does not list, as we found in our look at Ollama MTP.
MTP acceptance rates on an M4 Pro: Qwen3.6 35B-A3B and Qwen3.8 27B measured
On 11 September we ran both Qwen models through Hermes’s llama.cpp engine with MTP on, during the ARC-AGI-3 board test in Qwen3.6 35B-A3B vs Gemma 4 31B on a Mac. The server prints a draft acceptance line for every request: the share of drafted tokens the target model kept, and the mean number accepted per step.
| Model and request | Drafted tokens accepted | Mean accepted length | Generation speed |
|---|---|---|---|
| Qwen3.6 35B-A3B, first request | 313 of 393, 79.6% | 2.59 | 21.2 tok/s, includes load |
| Qwen3.6 35B-A3B, 16,384 token run | 10,530 of 11,705, 90.0% | 2.80 | 45.8 tok/s |
| Qwen3.6 35B-A3B, a later run | 10,804 of 11,157, 96.8% | 2.94 | 45.8 to 54.0 tok/s across its runs |
| Qwen3.8 27B, runs that hit the cap | 67% to 81% | 2.34 to 2.61 | 8.3 to 10.6 tok/s |
| Qwen3.8 27B, two runs that stopped early | 94.4% and 97.6% | not recorded | 8.3 to 10.6 tok/s |
Two things stand out. Acceptance on Qwen3.6 was high enough that each verification step kept close to three tokens on average. Qwen3.8 accepted less, and even with MTP on it generated fewer tokens a second than Gemma 4 31B did without any draft.
We did not run either Qwen model with MTP switched off on this machine, so we cannot say how much faster MTP made them. A speed-up figure needs that second run, and we will not estimate one.
High acceptance also needs care in reading. A draft is easy to accept when the next tokens are predictable, and text that repeats itself is predictable. The 96.8% Qwen3.6 run came from a test in which we later found one capped run looping, with 99.5% of the last quarter of its reasoning repeating earlier text. Acceptance measures how often the guesses matched, not whether the text was worth writing.
Why a GGUF file decides whether you get MTP at all
MTP only works if the draft layers survive into the file you run, and they do not always. When we read seven builds of Qwen3.5-35B-A3B from three publishers, one publisher’s files carried 41 blocks and another’s carried 40. The extra block held the four MTP tensors, stored at 8-bit precision, as covered in two GGUF builds of the same model differ by a whole block. A build without that block cannot use draft-mtp, whatever the flag says.
Gemma 4 packages its draft the other way. For Gemma 4 12B the draft is a separate four-block model with its own architecture name, gemma4-assistant, and none of the three main builds we checked contained it, which we set out in where the speculative decoding head ships. The same split shows up in Nemotron, whose 4-bit GGUF we examined for its MTP layers.
That leaves three checks before counting on MTP. The model family has to train a draft head, the GGUF build has to keep it, and the runtime has to be told to use it. Hermes’s catalog downloads Qwen3.6 from an unsloth repository with MTP in its name, unsloth/Qwen3.6-35B-A3B-MTP-GGUF, which is the only sign in a listing that the draft layers are there.
The measurement still missing on our machine is the plain one: the same Qwen3.6 prompt with --spec-type none, which would turn the acceptance rates above into a speed-up.