NVFP4 or q4_K_M in Ollama?

Ollama users who pull a Gemma 4 12B model today get two 4-bit options: q4_K_M at roughly 7.5 GB or nvfp4 at roughly 7.8 GB. The file size is nearly identical. Ollama says the quality is not. According to Ollama’s June 11, 2026 benchmark, NVFP4 roughly halves the perplexity gap between the q4_K_M quant and the unquantized BF16 baseline on the Gemma 4 12B model. For a business standardising local models across its machines, the format also comes with a platform catch.

NVFP4 is a 4-bit floating-point format introduced with NVIDIA Blackwell GPUs. Its purpose is to shrink models by roughly 4x relative to FP16 while preserving more accuracy than other 4-bit methods. Ollama added NVFP4 support through its MLX engine in a preview release in March 2026, then shipped production-ready performance in the June 2026 update of the MLX backend.

This piece covers what the format does, how it compares to what most Ollama users run today, and where it still falls short. The short answer, on the published record: Ollama’s quality claim comes with no figures, the one independent comparison favours q4_K_M, the files are no smaller, and inside Ollama the format runs only on a Mac. We have not yet run the two formats side by side ourselves, and every speed and quality figure below is someone else’s published measurement, attributed where it appears.

What does NVFP4 do differently from standard 4-bit?

Standard 4-bit quantization converts each weight into a small integer, then stores a shared scale factor for every block of values. The block size and the precision of that scale factor determine how much information survives the conversion. NVIDIA’s NVFP4 format uses a block size of 16 values, half the 32-value block used by its predecessor MXFP4. The scale factor itself is stored as an FP8 (E4M3) value instead of a power-of-two integer, which lets it represent fractional scales instead of only powers of two.

The result is a two-level scaling system. Each micro-block of 16 values gets an FP8 local scale. A second FP32 scale applies per tensor. Together, these layers recover dynamic range that uniform integer quantization loses. For a model weight distribution where values cluster near zero with occasional outliers, which is how most LLM weights actually behave, this two-level approach maps the distribution more accurately than a single integer scale over a larger block.

NVIDIA’s Blackwell Tensor Cores support NVFP4 natively in hardware, with mixed-precision execution that accumulates in FP16. On Apple Silicon, Ollama’s MLX backend runs the same arithmetic in Metal kernels. The format itself is the same in both cases, which means a model quantized for datacenter inference on Blackwell hardware can run on a MacBook.

What values can an NVFP4 element hold?

NVFP4 stores each element as E2M1: one sign bit, two exponent bits, one mantissa bit. NVIDIA’s TensorRT documentation writes it as S1E2M1. Its sixteen bit patterns decode to exactly this:

0, +/-0.5, +/-1, +/-1.5, +/-2, +/-3, +/-4, +/-6

That is sixteen encodings but fifteen distinct numbers, because +0 and -0 are separate bit patterns naming the same value. The exponent bias is 1. The value 0.5 is the format’s only subnormal. Normals follow (-1)^s x 2^(exp-1) x (1 + m/2). There is no infinity and no NaN. NVIDIA’s quantization library states the table as a literal in nvfp4_tensor.py:

E2M1_MAX = 6
e2m1_values = [0, 0.5, 1, 1.5, 2, 3, 4, 6, 0, -0.5, -1, -1.5, -2, -3, -4, -6]

The NVFP4 introductory blog post lists “0.0, 0.5, 1.0, 1.5, 2, 3, 4, 6” and describes the range as approximately -6 to 6, and the TensorRT quantized-types documentation gives the quantized value range as [-6, 6].

A claim keeps circulating that NVFP4 covers -8 to 7.5. It is wrong. The -8 is INT4, whose range is [-8, 7], and both ranges appear on the same TensorRT page. Our reading is that INT4’s upper bound picked up a half step from E2M1’s grid; that is an inference from where the two ranges sit in the documentation, not a traced origin. We could not find a published page that states the claim; it circulates as a question. The claim also fails its own arithmetic: -8 to 7.5 in steps of 0.5 is 32 values, not 16, while -8 to 7 in steps of 1 is 16 exactly, which is INT4. No four-bit format, and no floating-point format we could find at any width, has a maximum magnitude of 7.5.

A raw element capping out at 6 would be uselessly small for real weights. The scaling is the whole design:

Sixteen elements at four bits each, plus one eight-bit scale, is 72 bits per 16 weights: 4.5 bits per weight, plus one negligible per-tensor float. NVIDIA’s quantizer computes the global scale as amax(tensor) / (E2M1_MAX x E4M3_MAX), which is amax / (6 x 448), so every per-block scale is guaranteed to fit in E4M3. The global scale handles dynamic range and the block scale handles local precision. The element range is a grid, not a limit.

The row that matters is grid spacing. NVFP4’s values crowd near zero, where the steps are 0.5, and spread out at the top, where the step from 4 to 6 is 2. Model weights cluster near zero, so that grid fits them better than an even one at the same bit width.

How does NVFP4 differ from MXFP4?

The two formats use the identical element. Everything that separates them is metadata.

A smaller block means one outlier contaminates fewer neighbours. NVIDIA’s framing: “By reducing the block size from 32 to 16 elements compared to MXFP4, NVFP4 minimizes the influence of outliers and enables more precise scaling.” The scale format is the bigger difference. E8M0 has zero mantissa bits, so an MXFP4 scale can only be a power of two; if a block’s ideal scale is 1.5x, MXFP4 must round to 1x or 2x. NVIDIA again: MXFP4 “is limited to power-of-two scale factors (E8M0) and prone to high rounding errors.” NVFP4 pays 0.25 more bits per weight for that, 72 bits per 16 elements against 136 per 32.

NVIDIA’s NVFP4 pretraining paper reports that MXFP4 matches NVFP4 loss when trained on 36% more tokens, 1.36T instead of 1T, with relative training-loss error of roughly 2.5% for MXFP4 against 1.5% for NVFP4. That is NVIDIA measuring its own format, and the 36% comes from an ablation at a 1T-token budget, not from the paper’s headline run (a 12B model on 10T tokens, described as the longest publicly documented 4-bit training run). The headline run also needed Random Hadamard transforms, 2D quantization, stochastic rounding and selective high-precision layers.

MXFP4 exists anyway because it is the standard. It comes from the Open Compute Project’s Microscaling Formats v1.0 spec, backed by AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm. NVIDIA signed that standard and then shipped a format that, in its own words, “departs from the OCP MX standard in two ways.” We took the MXFP4 parameters from NVIDIA’s blogs and AMD’s ROCm documentation, which agree with each other; we did not read the OCP spec PDF directly.

How does NVFP4 quality compare to q4_K_M?

The headline number from Ollama’s benchmark is straightforward. On perplexity for Gemma 4 12B, Ollama says the gap between q4_K_M and unquantized BF16 is roughly twice the gap between NVFP4 and BF16; in its words, NVFP4 “roughly halves the quality loss of 4-bit quantization, relative to unquantized BF16.” The conditions given are Gemma 4 12B on a MacBook Pro M5 Max, averaged over 10 runs, with an 8,300-token input.

The post contains no perplexity value, no dataset name, no BF16 baseline figure and no q4_K_M figure. “Halves the quality loss” is a ratio with both terms withheld, so as published it cannot be checked, and we found no replication of it in Ollama’s repository or any discussion thread.

Perplexity is a coarse metric in any case. It measures how well the model predicts the next token in a held-out test set, which correlates with general output quality but does not capture every dimension of model behavior. On coding, reasoning, and instruction-following tasks, the gap may be narrower or wider depending on the model and the prompt.

What does the independent measurement show?

The one substantive comparison we found is llama.cpp Discussion #23853, in a thread opened by michaelw9999 (author of advanced-gguf-quantizer). The table was measured by mengqin, 3 to 5 June 2026, on Qwen3.6-27B:

Q4_K_M has less than half the KL divergence of the best NVFP4 build in the table, and roughly 2.6x less than the worst. On same top-p, how often the quantized model’s most likely token matches the reference, Q4_K_M holds 94.33% against 90.55 to 91.19%. KL divergence matters more than raw perplexity here: it measures how far the whole output distribution has drifted from the unquantized original, and for agent work, where a single wrong token derails a tool call, that drift is the number to watch.

The caveats stay attached. The model is Qwen3.6-27B, not the Gemma 4 12B Ollama measured. These are GGUF NVFP4 builds, a different code path from Ollama’s MLX NVFP4. And the NVFP4 builds are community conversions, not NVIDIA’s own. The same thread reports tg128 at 63.36 t/s for Q4_K_M against 71.57 t/s for the RSF build, but hardware is stated only loosely as an RTX 5090 with 64 GB, and the thread does not confirm that machine produced the speed figures, so we would not lean on that row.

What have NVIDIA and Red Hat published?

Both benchmark NVFP4 against BF16 and FP8 only. Neither has published a Q4_K_M or GGUF comparison. NVIDIA’s table for Gemma 4 26B-A4B:

That is near-BF16 accuracy at 4 bits on NVIDIA’s own terms. It does not answer the question a local user is asking. NVIDIA also reports 1% or less accuracy degradation for DeepSeek-R1-0528 moving from FP8 to NVFP4, at roughly 1.8x the memory saving, or about 3.5x against FP16.

Red Hat’s NVFP4 evaluation, published in February 2026, reports accuracy recovery against BF16 of about 99% at 70B to 235B, 97 to 99% around 30B, and 95 to 98% at 7B to 14B, with storage 1.5 to 1.8x smaller than FP8. Recovery was strongest at larger model scales. It states that throughput and latency analysis was “actively being finalized”, and we found no follow-up.

How much faster is NVFP4, and are the files smaller?

Ollama’s June 2026 tests say NVFP4 generates roughly 20 percent more tokens per second than the equivalent q4_K_M model on the updated MLX backend, averaged over ten runs with an 8,300-token input prompt. Ollama credits two sources for the speedup. MLX’s just-in-time compiler fuses multiple operations into single Metal kernels, which reduces GPU dispatch overhead, and Ollama reworked its GPU-backed sampling to run more efficiently.

Ollama’s own claim, as published: about 20 percent faster than q4_K_M at nearly the same file size. The measurements below do not reproduce it.

Ollama’s June update also introduced a snapshot system for agent workloads. It saves model state at branching points in multi-turn conversations, so when an agent hands off to a subagent or a user retries a response, the engine resumes from the saved state instead of reprocessing the entire context. This works with any model, not just NVFP4.

What do published speed measurements show?

The Apple Silicon result was reported by johnkea on 13 May 2026 and closed as a duplicate of #16030. It is the MLX path, on the platform NVFP4 is served on in Ollama, losing by 5%. The Blackwell figures come from knoopx’s Qwen3.6-35B-A3B-NVFP4-GGUF model card on Hugging Face, and the comparison there is against Q4_K_XL, not Q4_K_M. NVFP4 does win prefill on Blackwell, which is consistent with a format designed for tensor-core GEMM throughput: prefill is compute-bound, decode is bandwidth-bound, and a format that does not shrink the weights cannot help bandwidth.

If you are on Gemma 4, Ollama has also used MTP speculative decoding by default since v0.31.1, a much larger speed effect than the format choice, covered in Ollama MTP. Any NVFP4 against q4_K_M timing on Gemma 4 needs MTP held constant, with draft_num_predict: 0, or it measures the wrong thing.

Why are NVFP4 files not smaller?

On paper this should be a size win. Pure Q4_K’s source comment in llama.cpp’s ggml-common.h says “Effectively 4.5 bits per weight”: 2 + 2 + 12 + 128 = 144 bytes per 256 weights. But Q4_K_M is not 4.5. From llama.cpp’s quantize README, on Llama-3.1-8B, Q4_K_M is 4.8944 bpw (4.58 GiB), Q4_K_S is 4.6672 and Q6_K is 6.5625. The _M suffix is a mixture, promoting sensitive tensors, typically attn_v and ffn_down, to Q6_K. So NVFP4 at 4.5 should undercut Q4_K_M at 4.89 by about 8%. In shipped files it does not.

The Ollama sizes come from its library tags. A nominally 4.5-bit format shipping at 5.43 to 5.60 bits per weight is the size story in one number.

The reason is documented, and it is not a bug: conversion is partial by design. NVIDIA’s own recipe on the nvidia/Gemma-4-26B-A4B-NVFP4 model card (released 30 April 2026, Model Optimizer 0.43.0) is named nvfp4_experts_only. Only the MoE expert parameters are quantized, and everything else stays wide. knoopx’s card says the same from the other side: NVFP4 is applied “on all Linear weight layers, preserving auxiliary tensors (norms, SSM convolutions, router gates) in BF16/F32.” michaelw9999, in llama.cpp Discussion #23627, on the conversions generally: “In many cases it’s just leaving tensors as BF16 when they could be perfectly fine as Q8, or even less.”

There is a correctness question too. llama.cpp Discussion #22042, opened by ORippler on 17 April 2026, explains NVFP4 as a “derived tensor” needing a per-tensor F32 scale times a per-block F8 scale, F32_act = F4 x F8 x F32. ORippler’s assessment: the “current quantize_nvfp4/dequantize_row_nvfp4 functions inside GGML are incorrect when F32 != 1.0″, and the dequantization responsibility is “shared between GGML and its callee… undocumented in GGML and is, in my eyes, error prone.” We have not run gguf-dump on any of these files to count the NVFP4 tensor fraction ourselves, and we found no published count.

Which models support NVFP4, and how do you run one in Ollama on a Mac?

You need an Apple Silicon Mac and Ollama v0.19.0 or later, the release that shipped the MLX engine on 27 March 2026. Ollama’s MLX launch guidance was “a Mac with more than 32GB of unified memory”, though our read of the source found routing decided by model format, not by how much memory the Mac has. Check your version with ollama --version.

Confirmed NVFP4 tags in the Ollama library:

ollama run gemma4:12b-nvfp4
ollama run gemma4:26b-nvfp4
ollama run gemma4:31b-nvfp4
ollama run gemma4:e2b-nvfp4
ollama run gemma4:e4b-nvfp4

ollama run qwen3.5:4b-nvfp4
ollama run qwen3.5:9b-nvfp4
ollama run qwen3.5:27b-nvfp4
ollama run qwen3.5:35b-a3b-nvfp4
ollama run qwen3.5:35b-a3b-coding-nvfp4

Qwen3.6 NVFP4 tags also exist, qwen3.6:27b-coding-nvfp4 and qwen3.6:35b-a3b-coding-nvfp4, though we have seen them referenced in issue threads rather than confirmed on a library page. If you read that Ollama has only one NVFP4 model, that was true in March and is out of date.

The Gemma 4 26B NVFP4 model had seen over 4.3 million downloads by mid-June 2026. Gemma 4 is a general-purpose model with vision and tool-use capabilities, and the 4-bit variant makes it runnable on a 48 GB Mac Studio that would struggle to fit the BF16 version.

You can also convert your own models. The --quantize nvfp4 flag in Ollama’s import pipeline, first behind --experimental, graduated out of experimental status in April 2026. One caveat surfaced in an Ollama GitHub issue that month: the Qwen 3.6 35B-A3B model’s K-projection weights collapsed to zero during NVFP4 quantization when every value in a row fell below the smallest representable codepoint. The fix keeps certain attention projections in BF16 while quantizing only the MoE expert weights, and Ollama’s packaging tool was updated to mirror Red Hat’s mixed-precision approach. Users who pulled the model after the fix got the corrected version.

What are the gotchas with Ollama’s NVFP4 tags?

The -coding- tags are not fine-tunes. Per Ollama collaborator rick-github, they are “otherwise unmodified” models with different default sampling parameters, which you can set yourself on any tag.

The NVFP4 coding tags have no vision, “because the MLX backend they run on is not yet vision capable”. If your pipeline sends images, that is a hard blocker.

Structured output has been silently ignored on the MLX engine. Issue #17013, 2 July 2026, recorded format being dropped without an error, so a JSON schema is discarded and prose comes back. Verify structured output explicitly before trusting it in a pipeline.

The tag does not mean the whole model is NVFP4, and the file will not be smaller, for the reasons above. Run ollama list and ollama ps to confirm the size on disk and the loaded model before planning memory.

Why does Ollama NVFP4 only run on macOS?

NVFP4 in Ollama currently runs only on macOS via the MLX backend. Pull an NVFP4 tag on Windows or Linux, even with a Blackwell card installed, and the registry refuses:

412: this model requires macOS

The name causes the confusion. Blackwell’s fifth-generation tensor cores consume the format directly, handling element grouping, dynamic scaling and the 4-bit matrix multiply without a dequantization step, and Hopper cannot. But NVFP4 is also just a layout: four-bit E2M1 elements, an FP8 E4M3 scale for every 16 of them, an FP32 scale for the tensor. Any framework willing to write kernels can read it, and Apple’s MLX did, with Metal kernels and no NVIDIA silicon involved. Blackwell makes NVFP4 fast. It is not what makes it possible.

The commits show the path. mlx: add mxfp4/mxfp8/nvfp4 importing (Ollama PR #15015) merged on 24 March 2026. Three days later v0.19.0 shipped the MLX engine “in preview”, and by the launch post on 30 March Ollama was advertising ollama run qwen3.5:35b-a3b-coding-nvfp4. mlx: Support NVIDIA TensorRT Model Optimizer import landed on 27 April. v0.30.6 on 5 June added: “MLX embedding layers now use NVFP4 global scale for improved quantization on Apple Silicon.” The MLX performance post followed on 11 June.

rick-github has stated the restriction directly. On issue #16056, 9 May 2026: “NVFP4 support is provided by the MLX engine. The MLX engine currently only runs on macOS, so macOS is required to run an NVFP4 model. Support for Linux and Windows in the MLX engine is a work in progress.” Again on #15854, 15 May: “Transitively, running qwen3.6:27b-coding-nvfp4 requires macOS.” The restriction is enforced at the model catalogue, not locally. Users on #15854, #15772 and #16019 all hit the same 412. One user on #16265 installed ollama-windows-amd64-mlx.zip, got the MLX engine running on Windows, and was still refused, because the gate is an operating-system header on the catalogue request. A collaborator replied: “MLX support has not been officially announced for non-macOS environments.” That issue was opened on 22 May and remains open. Ollama’s issue tracker also has an open thread about adding platform annotations to model tags.

Is there a route to NVFP4 off macOS?

There is a separate path, and conflating it with the first is the easy mistake. llama.cpp added NVFP4 as a GGML type in PR #19769, merged 11 March 2026, CPU-only at merge with Metal, CUDA and Vulkan deferred. Native CUDA kernels followed in PR #20644, landing around build b8967 in late April, and Ollama bundles llama.cpp. So third-party NVFP4 GGUFs run on Ollama’s llama.cpp engine on hardware Ollama’s own NVFP4 tags refuse. Issue #16056 was opened after a GGML_ASSERT(type >= 0 && type < GGML_TYPE_COUNT) failed on a Gemma 4 26B NVFP4 model and asks for the gap to be closed. On current evidence, do not expect a speed win on Blackwell through that path.

If you are buying a card for NVFP4 specifically, the software lagged the silicon. Consumer and workstation Blackwell is SM 12.0, and SM 12.0 support trailed the datacentre SM 10.x parts in vLLM and CUTLASS. Outside Ollama, NVFP4 models are available on Hugging Face from Red Hat and NVIDIA, and vLLM, TensorRT-LLM and SGLang support NVFP4 inference.

The likelier route for Ollama runs through MLX: MLX has a CUDA backend upstream, and Ollama has been wiring it. PR #16871, “mlx: update and fix CUDA JIT packaging”, merged 23 June. PR #16894, “mlx: route fast quantized matmul on CUDA”, is open. If that lands, the 412 restriction stops making sense.

Should a business standardise on Ollama NVFP4 models?

Not on the record as it stands. Two facts come before quality for most teams. Ollama NVFP4 tags run only on macOS, and a Linux or Windows machine that asks for one gets the 412 error whatever GPU it has. A company with a mixed fleet, or one that tests on Macs and deploys on Linux servers, cannot use one model tag everywhere. The community GGUF route works on those machines, but it is a different file from a different conversion on a different engine, which defeats the point of a standard.

The usual reasons to accept a Mac-only format are missing too. The 26B tags are the same 18 GB, so there is no disk or download saving. The coding tags have no vision. The quality case rests on Ollama’s own claim without figures, and the only independent comparison favours Q4_K_M. For a mixed fleet, the format that runs on every machine is a Q4_K_M build on the llama.cpp engine; NVFP4 is then a Mac-only option whose value is a question for a team’s own test.

The MXFP4 question looks different for a business that runs models rather than trains them. The 36% figure is a training cost a downloader does not pay. What it does pay is memory: NVFP4’s extra 0.25 bits per weight is about 31 MB per billion parameters (0.25 x 1,000,000,000 / 8 = 31,250,000 bytes), so a 30-billion-parameter model would carry roughly 0.94 GB more in NVFP4, before counting any layers kept at higher precision. MXFP4 is the multi-vendor standard; NVFP4 is tied to Blackwell for native speed and, in Ollama, to Macs. We have not measured matching MXFP4 and NVFP4 builds of one model, so treat both as options to test rather than upgrades.

The element range is a quick check on a source. No buying decision turns on whether an element tops out at 6, but a supplier or write-up that quotes -8 to 7.5 has mixed NVFP4 with INT4, and its other claims about quantization deserve the same scrutiny. Ask instead for size on disk and quality against a named baseline on the same model.

What should a team test before switching?

Pick one task staff already do, such as pulling fields from supplier invoices, and gather a fixed set of documents whose right answers you already hold. Run the same model in both formats in Ollama on the same Mac, with MTP held off, and record four things for each:

• the share of fields extracted correctly against your known answers
• seconds per document, from request to last token
• tokens per second at the context length you actually use
• peak memory while the batch runs

The first is the one a switch would have to justify itself on, since the file is the same size. For an automated pipeline, add the checks a chat session forgives. Send the pipeline’s real JSON schema with format and confirm the reply validates. Stop if any step sends images. Pin the exact tag and the Ollama version in the pipeline’s configuration, because the engine, the MTP default and the list of tags have all changed between releases this year. And do not depend on the tags if the pipeline may later move to a Linux server.

We have not run any of this on our own machine, so this piece gives no speed, memory or quality figure of its own and no verdict on standardising. The run is on our list: the same model as an NVFP4 tag and a q4_K_M tag in Ollama on our M4 Pro, with a tensor audit, size on disk, peak memory, tokens a second at 8K and 32K context, perplexity against a BF16 baseline, a structured-output check and a task benchmark on documents with known answers, all with draft_num_predict: 0 so MTP is held constant.

Against BF16, NVFP4 is a good format on NVIDIA’s and Red Hat’s numbers, and on Blackwell tensor cores the no-dequantization path is an advantage GGUF cannot match. The version reaching local users through Ollama is a partial conversion that does not shrink files and has not beaten Q4_K_M in any published measurement we could find. For where MLX and llama.cpp diverge on a Mac more generally, see MLX vs llama.cpp vs MetalRT.