This is a research piece. We have not run these comparisons ourselves. Everything below is other people’s published measurement, attributed, with the gaps in the record named as gaps. We say where we think the evidence points and we do not dress it up as our own testing.
The headline: the only party claiming NVFP4 beats Q4_K_M is Ollama, and it published no numbers to check. The one independent perplexity comparison we could find has Q4_K_M winning on quality by roughly 2.6x on KL divergence. Both published speed comparisons have NVFP4 losing on decode. And the files are not smaller.
For what the format is and why anyone reached for it in the first place, start with what Ollama NVFP4 means for local model quality. This piece is narrower: it audits the one comparison that decides whether you should switch.
The claim, and what is missing from it
From Ollama’s MLX performance post, 11 June 2026:
> NVFP4 “roughly halves the quality loss of 4-bit quantization, relative to unquantized BF16.”
Conditions given: Gemma 4 12B, MacBook Pro M5 Max, averaged over 10 runs, 8,300-token input. The speed claim alongside it is “about 20% faster than q4_K_M on the updated engine.”
What the post does not contain: any perplexity value, any dataset name, any BF16 baseline figure, any Q4_K_M figure. “Halves the quality loss” is a ratio with both terms withheld. As published it cannot be verified or falsified, and we could find no replication of it anywhere, not in Ollama’s repo, not in any discussion thread.
That is not an accusation of dishonesty. It is a statement about what a reader can actually do with the claim, which is nothing.
The independent measurement
The one substantive comparison we found is llama.cpp Discussion #23853, in a thread opened by michaelw9999 (author of advanced-gguf-quantizer). The table below was measured by mengqin, 3-5 June 2026, on Qwen3.6-27B:
| Variant | Mean PPL | Mean KLD | RMS delta-p | Same top-p | Size |
|---|---|---|---|---|---|
| Unsloth Q4_K_M (baseline) | 6.9368 +/- 0.0457 | 0.02235 +/- 0.00064 | 4.04% | 94.33% | 15.92 GiB (5.01 BPW) |
| advanced-gguf-quantizer RSF | 7.030 | 0.0446 | 5.77% | 91.92% | 15.27 GiB (4.80 BPW) |
| Jianqiao NVFP4 v1 | 7.061 | 0.0514 | 6.17% | 91.19% | 15.53 GiB (4.88 BPW) |
| michaelw9999 NVFP4 v1 | 7.117 | 0.0582 | 6.67% | 90.55% | n/a |
Q4_K_M has less than half the KL divergence of the best NVFP4 build in the table, and roughly 2.6x less than the worst. On “same top-p”, how often the quantized model’s most likely token matches the reference, Q4_K_M holds 94.33% against 90.55-91.19%.
KLD matters more than raw perplexity here. Perplexity tells you how the model scores a corpus; KL divergence tells you how far its whole output distribution has drifted from the unquantized original. For agent work, where a single wrong token derails a tool call, distribution drift is the number to watch.
Three caveats we will not bury. The model is Qwen3.6-27B, not the Gemma 4 12B Ollama measured. These are GGUF NVFP4 builds, which is a different code path from Ollama’s MLX NVFP4. The comparison is not apples to apples with the vendor claim. And the NVFP4 builds are community conversions, not NVIDIA’s own.
The same thread reports tg128 at 63.36 t/s for Q4_K_M against 71.57 t/s for the RSF NVFP4 build, a speed win in the other direction. Hardware is stated only loosely as an RTX 5090 with 64 GB, and the thread does not confirm that machine produced the speed table, so we would not lean on that row.
Speed, where it has been measured
| Platform | NVFP4 | Q4_K variant | Source |
|---|---|---|---|
| MacBook Pro M3, MLX | gemma4:31b-nvfp4 6.29 tok/s | gemma4:31b-it-q4_K_M 6.62 | Ollama issue #16127 |
| Blackwell, GGUF (decode) | 136 t/s | Q4_K_XL 188 t/s | knoopx model card |
| Blackwell, GGUF (prefill) | 9767 t/s | Q4_K_XL 9121 t/s | same |
The Apple Silicon result is the awkward one for the vendor claim: it is the MLX path, on the platform NVFP4 was optimised for, losing by 5%. Reported by johnkea on 13 May 2026, closed as a duplicate of #16030. Why that path is the only one Ollama will serve the format on is its own story, told in NVFP4 on Apple Silicon.
NVFP4 does win prefill on Blackwell. That is consistent with a format designed for tensor-core GEMM throughput. Prefill is compute-bound, decode is bandwidth-bound, and a format that does not shrink the weights cannot help bandwidth.
Why the bit budget does not deliver
On paper this should be a size win. Pure Q4_K’s own source comment in llama.cpp’s ggml-common.h says “Effectively 4.5 bits per weight”: 2 + 2 + 12 + 128 = 144 bytes per 256 weights. NVFP4 is 4.5 bpw exactly: 16 elements at 4 bits plus an 8-bit E4M3 scale is 72 bits per 16, a budget we derive in full in the corrected NVFP4 value table.
And Q4_K_M is not 4.5 bpw. From llama.cpp’s own quantize README, on Llama-3.1-8B: Q4_K_M is 4.8944 bpw (4.58 GiB), Q4_K_S is 4.6672, Q6_K is 6.5625. The _M suffix is a mixture, promoting sensitive tensors, typically attn_v and ffn_down, to Q6_K.
So NVFP4 at 4.5 should undercut Q4_K_M at 4.89 by about 8%. In shipped files it does not:
| File | Size | BPW |
|---|---|---|
gemma4:26b-a4b-it-q4_K_M (Ollama) | 18 GB | n/a |
gemma4:26b-nvfp4 (Ollama) | 18 GB | n/a |
| Unsloth Gemma-4-26B-A4B Q4_K_M | 16.9 GB | n/a |
| ggufbench Gemma-4-26B-A4B NVFP4 | ~16.45 GiB | 5.60 |
| knoopx Qwen3.6-35B-A3B NVFP4 | 20.8-21 GB | 5.43 |
A nominally 4.5-bit format shipping at 5.43-5.60 bits per weight is the whole story in one number.
The mechanism: conversion is partial by design
This is documented, and it is not a bug.
NVIDIA’s own recipe is named nvfp4_experts_only. On the nvidia/Gemma-4-26B-A4B-NVFP4 model card (released 30 April 2026, Model Optimizer 0.43.0), only MoE expert parameters are quantized. Everything else stays wide.
Community builds say the same from the other side. knoopx applies NVFP4 “on all Linear weight layers, preserving auxiliary tensors (norms, SSM convolutions, router gates) in BF16/F32.”
There is a correctness dimension too. llama.cpp Discussion #22042, opened by ORippler on 17 April 2026, explains NVFP4 as a “derived tensor” requiring a per-tensor F32 scale times a per-block F8 scale, F32_act = F4 x F8 x F32. ORippler’s assessment: the “current quantize_nvfp4/dequantize_row_nvfp4 functions inside GGML are incorrect when F32 != 1.0″, and the dequantization responsibility is “shared between GGML and its callee… undocumented in GGML and is, in my eyes, error prone.”
And michaelw9999, in llama.cpp Discussion #23627, on the conversions generally: “In many cases it’s just leaving tensors as BF16 when they could be perfectly fine as Q8, or even less.”
We have not run gguf-dump on any of these files to count the NVFP4 tensor fraction ourselves, and we could find no published count. If you see a specific ratio quoted anywhere, ask for the source. We looked and the record is silent.
What the format vendors actually published
Both NVIDIA and Red Hat benchmark NVFP4 against BF16 and FP8 only. Neither has ever published a Q4_K_M or GGUF comparison. That absence is itself informative: the format’s sponsors are positioning it against datacenter high-precision baselines, not against the llama.cpp quant family a local user would actually be choosing between.
NVIDIA’s table for Gemma 4 26B-A4B, NVFP4 against BF16:
| Benchmark | BF16 | NVFP4 | Delta |
|---|---|---|---|
| GPQA Diamond | 80.30 | 79.90 | -0.40 |
| AIME 2025 | 88.95 | 90.00 | +1.05 |
| MMLU Pro | 85.00 | 84.80 | -0.20 |
| LiveCodeBench pass@1 | 80.50 | 79.80 | -0.70 |
| IFBench | 77.77 | 78.10 | +0.33 |
| IFEval | 96.60 | 96.40 | -0.20 |
That is a genuinely strong result on its own terms, near-BF16 accuracy at 4 bits. It just does not answer the question a local user is asking.
Red Hat’s February 2026 write-up on accelerating LLMs with NVFP4 quantization reports accuracy recovery against BF16 of ~99% at 70B-235B, 97-99% around 30B, and 95-98% at 7B-14B, with storage 1.5-1.8x smaller than FP8. It states explicitly that throughput and latency analysis was “actively being finalized”, and we found no follow-up.
Where we land
On the published record, for a local user on a Mac choosing between an NVFP4 tag and a q4_K_M GGUF today:
- Quality: the only independent comparison favours Q4_K_M, materially, on KLD and top-p agreement. The contrary claim has no numbers attached.
- Decode speed: NVFP4 loses on both platforms measured.
- Prefill speed: NVFP4 wins on Blackwell. Unmeasured on Apple Silicon.
- Size: no win. Shipped NVFP4 files run 5.4-5.6 bpw.
- Against BF16: NVFP4 is genuinely good, per NVIDIA and Red Hat. That is a different comparison.
We are not saying NVFP4 is a bad format. The pretraining and inference results against BF16 are real, and on Blackwell tensor cores the no-dequantization path is a structural advantage that GGUF cannot match. We are saying that the version of NVFP4 currently reaching local users is a partial conversion that does not shrink files and has not beaten Q4_K_M in any published measurement we could find.
The comparison we would want, and which nobody has published: same model, same machine, NVFP4 against q4_K_M, with the tensor conversion ratio reported alongside, MTP held constant, and a task benchmark rather than perplexity alone. That is a gap in the record, not a gap in our reporting. And it is the one we intend to close on our own hardware. The setup for it is written up in running an NVFP4 model in Ollama on a Mac.
The short version
- Ollama claims NVFP4 “halves the quality loss” vs q4_K_M. No PPL values, no dataset, no baseline. Unverifiable as published, and unreplicated.
- The one independent comparison (llama.cpp Discussion #23853, Qwen3.6-27B): Q4_K_M KLD 0.0224 against NVFP4’s 0.0514-0.0582. Q4_K_M wins.
- Decode: 6.29 vs 6.62 tok/s on M3; 136 vs 188 t/s on Blackwell. NVFP4 loses both.
- Prefill on Blackwell: NVFP4 wins, 9767 vs 9121.
- Size parity, not advantage. 18 GB vs 18 GB; NVFP4 builds ship at 5.43-5.60 bpw against a nominal 4.5.
- Cause:
nvfp4_experts_only. Partial conversion by design, plus open GGML dequantization correctness questions. - NVIDIA and Red Hat only ever benchmark against BF16/FP8. No vendor has published an NVFP4-vs-Q4_K_M table.