MLX, llama.cpp or MetalRT on Apple Silicon?

Same 7B model, same M2 Ultra: MLX decodes about 53 percent faster than llama.cpp. MetalRT peaked higher still, but on a much smaller model.

Why does MLX have an M5 Neural Accelerator advantage?

MLX is Apple’s first-party machine learning framework, built from the ground up for the Metal graphics architecture. It launched in late 2023 and has since become the default inference path for users inside Apple’s ecosystem who want direct GPU access without third-party abstraction layers, as we covered in our Ollama vs MLX vs Jan comparison.

The M5 generation, released in late 2025, added dedicated Neural Accelerators. These are hardware units designed for the matrix-multiplication operations that dominate transformer inference. These units are accessible through MLX via the Metal 4 Tensor API. In benchmarks published by Apple Machine Learning Research (2025), a Qwen-2.5 7B model running on an M5 MacBook Pro showed up to 4x faster time-to-first-token compared to the same model on an M4 machine. Generation throughput improved 19% to 27%. The improvement tracks the M5’s memory bandwidth increase from 120 GB/s to 153 GB/s.

That paper’s comparison, which ran on an M2 Ultra, found MLX sustained the highest generation throughput across the models tested. The study is notable because it is a published head-to-head of these engines and includes full reproduction scripts. MLX reached approximately 230 tok/s on the Qwen-2.5 7B 4-bit test, while llama.cpp reached approximately 150 tok/s under identical settings.

The MLX advantage is clearest on Apple’s own hardware, especially M5 and later. The framework gets first access to new Metal features, and the model conversion tools are maintained by the same team that builds the inference engine. That vertical integration matters when a new chip ships and developers want accelerated inference on day one.

What did llama.cpp gain from the Metal Tensor API?

llama.cpp is the most widely deployed open-source inference engine for local LLMs. It supports more model formats, more quantization schemes, and more hardware backends than any alternative. On Apple Silicon, it has historically lagged MLX on raw throughput because it must map operations through the Metal Performance Primitives layer rather than calling Apple’s Metal APIs directly.

That gap narrowed significantly between late 2025 and early 2026. The llama.cpp project merged support for the Metal 4 Tensor API in PR #16634. The change gates Neural Accelerator access to M5 and later chips. It reworked how llama.cpp performs matrix-matrix multiplication on Apple GPUs and unlocked hardware paths that were previously unavailable to it.

The impact showed up in PR #20962, merged in March 2026. A benchmark on an M5 Max with 40 GPU cores and 64 GB of memory measured LLaMA 7B Q4_0 prompt processing at 3246 tokens per second. That is a 229% improvement over the same build before the Tensor API optimizations. Text generation, or decode, improved more modestly from 102 to 110 tok/s. This confirms that decode remains memory-bandwidth bound rather than compute bound on this engine.

The same pull request showed what the M5 upgrade buys. Prompt processing for Mistral 8B Q8_0 on an M4 Max ran at 631 tok/s. On an M5 Max, it ran at 2695 tok/s, a 4.27x improvement. A follow-up change in PR #19369 improved CPU and GPU interleaving during graph encoding. The improvement delivered 1% to 6% decode throughput gains across models from DeepSeek MoE to Gemma 3 to Qwen 3.

llama.cpp does not lead on raw decode speed against MLX or MetalRT. But its ecosystem reach, format support, and the speed of its recent Metal optimizations make it the engine to watch when the next round of Apple hardware ships. No other engine can run as many different model formats out of the box.

Does llama.cpp support MLX?

No. llama.cpp has never supported MLX, and there is no open work to make it. On Apple Silicon it runs on the GGML Metal backend, which people reasonably confuse with MLX. The answer sits in llama.cpp Discussion #4345, opened on 6 December 2023, four days after MLX was released. The maintainer slaren wrote that MLX “is too high level for what we need in ggml”, while noting its Metal backend might help them learn to optimise their own Metal kernels. MLX is a framework, with array operations, autograd and a whole programming model; GGML wants a backend that hands it kernels for a device. Adopting MLX would put a framework underneath a framework.

We searched every llama.cpp issue and pull request with MLX in the title. There were nine, all time. Every one is about conversion (convert_hf_to_gguf.py failing on MLX-fused checkpoints, GGUF to MLX and back) or about performance comparison. None proposes MLX as a backend, and there is no newer maintainer statement than slaren’s.

The question keeps coming back because users feel the gap. Issue #19366, filed on 5 February 2026, reported llama.cpp at roughly one third of MLX’s token generation on Qwen3-Coder-Next on an M4 Pro, with the gap worst on hybrid and linear-attention architectures. The stale bot closed it on 24 March 2026 with no maintainer response. We have not measured that model ourselves; it is a user’s report.

What does MetalRT do differently?

MetalRT is a newer entrant developed by RunAnywhere and released in early 2026. It is a dedicated Metal inference engine that skips the abstraction layers both MLX and llama.cpp sit on. It is not a fork of either project. It is a ground-up implementation that calls Metal directly for matrix operations and memory management.

The company published benchmarks in April 2026 comparing MetalRT against mlx-lm, llama.cpp, Ollama, and uzudil on an M4 Max with 64 GB of memory. The test used the same MLX 4-bit model files for MetalRT and mlx-lm. The shared model format gave a direct engine-to-engine comparison without a format variable. MetalRT’s peak decode speed reached 658 tok/s on Qwen3-0.6B. Against mlx-lm, it was 1.10x to 1.19x faster on decode across the model range. Against llama.cpp, the gap was 1.35x to 2.14x.

These numbers are strongest on decode, which is the bottleneck users feel most directly during chat and text generation. Each token comes out slightly faster, and over thousands of tokens the difference adds up to seconds of wall-clock time. MetalRT does not currently match MLX on time-to-first-token for large prompt sizes on M5 hardware. Its ecosystem is small. The engine supports a narrower range of model formats and has fewer community tools. But for decode-bound workloads on M4 and M5 hardware, it is the fastest open-source option measured so far.

How do the three engines compare by the numbers?

The table below consolidates available data across the three engines from published benchmarks.

The pattern is clear. MLX leads on prompt processing and time-to-first-token when running on M5 hardware, thanks to direct Neural Accelerator access. MetalRT leads on decode throughput on M4 and M5 machines, with a meaningful margin over both alternatives. llama.cpp has closed the gap substantially in the last six months, especially on prompt processing, but still trails on decode.

How do you choose an engine for your workload?

For interactive chat on an M5 MacBook Pro, MLX gives the fastest first-token response. The TTFT improvement from the Neural Accelerators means the model starts answering before the user notices a delay. For batch generation and long-form text output on the same hardware, MetalRT’s decode advantage matters more. Each token comes out faster, and the difference accumulates quickly.

For users on M4 or earlier hardware, the picture shifts. MLX still benefits from Apple’s framework optimizations, but the Neural Accelerator gap disappears. MetalRT shows its largest decode advantage on these chips. Large parts of the install base are on M4 or M3 machines, and for those systems MetalRT is the strongest option for decode-bound work.

For multi-user serving scenarios, the vllm-mlx project, described in a 2026 arXiv paper, achieved 21% to 87% higher throughput than llama.cpp on an M4 Max, with continuous batching scaling to 4.3x aggregate throughput at 16 concurrent requests. That makes MLX the current leader for server-like workloads on Apple Silicon.

Which engine does Ollama use on a Mac?

Both, which is where most of the confusion starts. Ollama bundles llama.cpp, so “Ollama supports MLX” and “llama.cpp supports MLX” sound like the same sentence, and they are not. Ollama shipped its MLX engine “in preview” in v0.19.0 on 27 March 2026, with a launch post on 30 March advising “a Mac with more than 32GB of unified memory.” In v0.30.0 on 13 May it reintroduced llama.cpp alongside MLX, describing it as something that “augments the MLX engine on Apple Silicon”. On a current Mac install, the model decides which engine serves a request.

Ollama’s published figures for its MLX engine, which are its claim and not our measurement, give Qwen3.5-35B-A3B on an M5 a decode rise from 58 to 112 tok/s and a prefill rise from 1154 to 1810 tok/s. Two things we could not confirm. “Preview” is the last official label for the MLX engine; the June performance post drops the word but never declares general availability. And third-party posts claim that below 32 GB you automatically get llama.cpp Metal instead of MLX; that rule appears neither in the v0.30.0 release notes nor in Ollama’s GPU documentation, which mentions Metal and says nothing about MLX, so we do not treat it as fact.

Which engine should a business standardise on?

That turns on what else the business runs more than on which engine is faster on one Mac. A team with Macs on desks and Linux or Windows machines elsewhere gets one engine everywhere only with llama.cpp: the same GGUF file, quantisation and server flags work on Mac, Windows, Linux, CUDA, ROCm, Vulkan and bare CPU, so a problem found on a laptop can be reproduced on a server. It also carries the whole GGUF ecosystem on Hugging Face, every quantisation scheme including imatrix quants and the q*_K family, and partial offload. MLX runs only on Apple hardware and uses safetensors files rather than GGUF.

A team that runs only Macs has a real choice, and newer options arrive on MLX first: in Ollama, NVFP4 tags and MTP speculative decoding live there (see NVFP4 on Apple Silicon, where the format is macOS-only because it arrived through MLX). The MLX surface is thinner. The coding NVFP4 tags have no vision, and Ollama issue #17013, on 2 July 2026, reported structured outputs via format silently ignored on the MLX engine. A schema dropped without an error can pass a quick manual check and still break a pipeline that expects JSON, and it sits close to the structured-output failure we hit through LM Studio below. We have not reproduced the Ollama issue on our M4 Pro; sending a format request to a model on Ollama’s MLX engine and checking the schema is the test, and it is on our list. Because one Ollama install holds both engines, pin the Ollama version and the exact model tag in each pipeline’s configuration, so a change of engine cannot arrive with a routine update.

How do MLX and llama.cpp compare on a team server with two users?

Since the research above was gathered, we have run MLX and llama.cpp against each other ourselves, through LM Studio on our test machine, an M4 Pro with 48 GB of memory. LM Studio’s MLX engine was version 1.11.0 and its llama.cpp engine 2.47.0, and every model was loaded alone. The full method is in our test of LM Studio parallel requests on MLX. MetalRT was not part of it.

Tokens written per second. StrideNote hands-on testing, 12 October 2026.

For a team server, the combined column is the one to read. On the dense Gemma 4 31B, llama.cpp gave two users half the speed each and no more total output than one user got; MLX gave both users close to full speed. On the mixture-of-experts Qwen3.6, both engines shared the work, and MLX shared it better. Reading a long prompt moved too: on Qwen3.6, a 20,033-token agent prompt took 74.8 seconds to the first token on llama.cpp and 30.3 on MLX, in our MLX vs llama.cpp run.

Before moving a shared server to MLX, check three things we hit. Tool calling held on both engines, eleven checks out of eleven. Structured output did not: the Qwen3.6 MLX build returned schema-bound JSON in the reasoning field and left the reply empty, five times out of five. And the MLX build ignored the requested context size, loading at 183,296 tokens when we asked for 32,768. We tested two users, not four, so the point where a dense model stops scaling on MLX is still unmeasured. Three and four users on each engine, and MetalRT and Ollama’s MLX engine under the same two-user test, would settle it, and they are on our list. Ollama now runs MLX too, as our piece on Ollama and MLX sets out, but we have not put it through this test.

All three engines are free and open source, and all three are under active development. The gap between them is narrower at the end of 2026 than it was at the start of the year, and it will keep shrinking as M5 adoption grows and optimizations propagate across all three codebases.

The best engine is not the one with the highest headline number. It is the one that fixes the bottleneck your workload hits first. For most users on M5 hardware doing interactive work, that means MLX for prompt processing and MetalRT for decode. For users on older hardware or with format compatibility needs, llama.cpp remains the most practical choice. Once the engine question is settled, our guide to which local LLM to run in 2026 helps you choose the model to feed it.

Sources: “Production-Grade Local LLM Inference on Apple Silicon” (arXiv, 2025), Apple Machine Learning Research “Exploring LLMs with MLX and M5” (2025), RunAnywhere “MetalRT Fastest LLM Decode Engine” (2026)

Disclaimer: The engine comparison in this piece is research: its figures come from an arXiv paper, Apple’s machine learning research, the llama.cpp project and RunAnywhere, each named where it appears. The one exception is the team-server section, where the MLX and llama.cpp figures are our own, measured through LM Studio on our M4 Pro on 12 October 2026. The Ollama MLX figures are Ollama’s own, and the llama.cpp issue figures are a user’s report. We have not run MetalRT.