Local inference on a Mac is straightforward until several requests overlap. Then time to first token, memory growth, and admission control become serving problems rather than model-execution problems. vllm-metal brings vLLM’s scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, with MLX and Metal handling execution.

v0.28.0 adds batched MTP, GGUF and hybrid-model support, and automatic prefill acceleration on M5. On SiliconBench’s agent split, vllm-metal keeps TTFT flatter as concurrency rises while serving from a fixed memory budget.

How vllm-metal fits into vLLM

vllm-metal plugs into upstream vLLM. vLLM provides the V1 scheduler, paged KV block management, chunked prefill, sampling, and the OpenAI-compatible frontend with streaming and tool-call parsing. mlx_lm provides the model implementations; MLX executes them.

At the model level, vllm-metal reuses mlx_lm’s weight loading, RMSNorm, linear, MoE, and MLP layers unchanged. Those layers process each token independently, so they run on a packed token axis without knowing request boundaries. Attention does need those boundaries, so vllm-metal replaces stock attention with a paged varlen Metal kernel. Most of the plugin’s model-specific code therefore sits in one layer.

Architecture overview: clients speak the OpenAI API to upstream vLLM's frontend and V1 scheduler, which hand the vllm-metal model runner a packed step plus block tables; the runner reuses mlx_lm's token-wise layers and adds custom Metal paths for paged varlen attention, MTP, and M5 NAX prefill, all executing through MLX and Metal on Apple Silicon unified memory

Start an OpenAI-compatible server

Install vllm-metal into its own virtual environment and activate it:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate

The installer adds the plugin, vLLM core, and their dependencies to ~/.venv-vllm-metal.

Then launch a model:

# --gpu-memory-utilization caps vLLM's share of unified memory; see below.
vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5

# 64 GB Macs: the 27B hybrid
# vllm serve mlx-community/Qwen3.8-27B-4bit --gpu-memory-utilization 0.7

# Speculative decoding: Gemma 4 with its MTP assistant
# vllm serve google/gemma-4-E4B-it --gpu-memory-utilization 0.5 \
#   --max-model-len 16384 --no-async-scheduling \
#   --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

More models: model matrix.

The server speaks the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-0.8B",
       "messages": [{"role": "user", "content": "Say hi"}]}'

Anything that takes an OpenAI-compatible base URL can point at http://localhost:8000/v1, coding agents included; the vLLM docs cover Claude Code and Codex setup.

Set a predictable memory budget

vllm-metal reserves its KV cache at startup and serves every request from that fixed pool. If you are used to runtimes whose memory footprint moves with load, this fixed pool is the main mental-model change. --gpu-memory-utilization sets the share of the Mac’s GPU memory budget used for serving.

Apple Silicon has no separate VRAM. GPU allocations come from the same unified memory used by macOS, your browser, and your editor. --gpu-memory-utilization is a serving budget; process memory can exceed it. As a starting point, 0.5 keeps a laptop usable while it serves; a dedicated machine can go higher.

The scheduler tracks the available KV pages, packs requests against that budget, and queues requests that do not fit. Under a burst, queue depth grows while the KV pool stays fixed.

Before sizing the KV pool, vllm-metal accounts for model weights and temporary buffers, keeping memory predictable as batch shapes change (PR #268).

Packed queries and paged KV

In mlx_lm’s padded batches, attention queries have shape [B, H, T_max, D]: every request gets the longest query length in the batch. MLX’s scaled_dot_product_attention has no varlen interface.

vllm-metal preserves vLLM V1’s unified model step for chunked prefill and decode. It packs every scheduled query token into [total_q, H, D], with cu_seqlens marking request boundaries, and runs the mixed step in one model forward.

KV is separate: mlx_lm keeps a contiguous [B, H, T, D] cache, while vllm-metal stores KV in fixed-size pages addressed by per-request block tables. Admitted requests can grow without reshaping a padded cache.

Only vllm-metal pairs a packed query axis with paged KV storage among the Apple Silicon serving stacks we audited:

Engine Encoding Query axes KV Batch Spec Decoding
mlx_lm padding [B, T_max] contiguous No
oMLX padding [B, T_max] contiguous No
llama.cpp mask [total_q] fixed cells Yes
vllm-metal cu_seqlens [total_q] paged Yes

A padded rectangle versus vllm-metal's packed varlen step for a 30,000 + 5,000 + 10 token batch, with KV read from paged storage

The first payoff is eliminating padded computation. In the figure’s 30,000 + 5,000 + 10-token step, padding sends 90,000 token rows through the model; packing sends 35,010. Because the packed axis runs through the whole forward pass, those extra rows disappear from attention, MLP, and MoE.

Padding also spends memory according to the longest sequence in the batch. A concurrent mix that fits comfortably on a large machine can push a smaller one into macOS memory compression and slow down without an explicit error. Sizing one paged pool up front removes that failure mode.

The second payoff is keeping ragged work in one batch. In a speculative step, one request may contribute a single decode token, another its last token plus a request-specific number of drafts, and another a prefill chunk. A [B, H, T_max, D] query tensor must either pad those rows to a common width or split them across forwards. vllm-metal instead concatenates the windows as [total_q, H, D] and verifies them in one target-model forward.

The Metal kernel ports vLLM’s unified Triton kernel, described in The Anatomy of a Triton Attention Kernel, to Apple GPUs, down to the binary search each threadgroup runs over cu_seqlens to find which request owns its query token.

Concurrent serving under agent load

Coding agents create concurrency by fanning out tool calls. Each call carries a few thousand tokens of context and returns a short reply, with several in flight at once. Every round trip pays TTFT before work can continue; end-to-end request latency sets the duration of the turn.

Qwen3.8-27B

We measured the agent split with SiliconBench, our benchmark suite for LLM inference engines on Apple Silicon. The workload contains 100 requests averaging 4.6K input and 70 output tokens, run closed loop at concurrency 1, 2, and 4 against Qwen3.8-27B in 4-bit on a 64 GB M5 Pro. Each engine runs the 4-bit build its own users would install, so weight footprints are not matched across engines: the MLX conversion carries 14.1 GiB of text weights against llama.cpp’s 15.3 GiB UD-Q4_K_M. Every concurrency level is an independent measurement: each one gets a fresh server, and oMLX additionally gets an empty cache directory, because its prefix cache lives on disk and outlives the process. oMLX appears twice because it is the only engine here whose prefix cache can live on disk: it spills KV cache to SSD without eviction, and ships with its in-memory tier switched off. We report it both that way and with the cache held in memory instead, which is what every other engine in the figure does.

SiliconBench agent split on Qwen3.8-27B: TTFT, end-to-end request latency, and output token throughput versus concurrency for llama.cpp, vllm-metal, and oMLX with its prefix cache in memory and on SSD

  • The two oMLX lines are one engine with its prefix cache in two different places. SSD offload is oMLX’s own default: reusable KV blocks go to disk, and its in-memory tier ships switched off. RAM cache keeps those blocks in memory and writes nothing to disk — what llama.cpp and vllm-metal already do. Neither carries anything from one concurrency level to the next.
  • Single-stream, oMLX with SSD offload is ahead: 7.0 s to first token against vllm-metal’s 7.3 s, and slightly better on latency and throughput too. The ordering inverts as soon as requests overlap, which is the case this scheduler is built for.
  • vllm-metal holds the flattest TTFT curve, rising only from 7.3 s to 9.3 s across a fourfold increase in concurrency. llama.cpp goes 11.4 s to 15.7 s; oMLX climbs to 32.9 s with its cache in memory and 38.9 s with SSD offload. End-to-end latency orders the same way: 36.1 s for vllm-metal at concurrency 4 against 56.9 s for llama.cpp, 46.4 s for oMLX in memory and 52.5 s with offload.
  • vllm-metal is the only engine whose output throughput rises at every step, 5.8 to 6.8 to 7.3 tokens per second. The others end within a few percent of where they started or fall outright, which is what a scheduler that cannot overlap prefill with decode looks like under load.
  • Where oMLX keeps its prefix cache matters more than whether it has one. Holding it in memory instead of on SSD is worth 14% output throughput at concurrency 4 (5.47 against 4.80 tok/s) and six seconds of TTFT (32.9 s against 38.9 s). Both arms see the same reuse — each concurrency level starts from an empty cache either way — so the difference is the cost of serving a hit, not the number of hits.

Gemma 4 E4B

Gemma 4 E4B is small enough to sweep through concurrency 16 on the same machine and agent split.

SiliconBench agent split on Gemma 4 E4B: TTFT, end-to-end request latency, and output token throughput versus concurrency for llama.cpp, vllm-metal with and without the MTP drafter, and oMLX with its prefix cache in memory and on SSD

At concurrency 16, vllm-metal schedules all requests together and reaches 2.3 s TTFT. llama.cpp’s four default slots leave twelve requests queued, pushing TTFT to 32.4 s, and oMLX reaches 14.1 s with its cache in memory and 22.9 s with SSD offload. The MTP arm turns the same continuous batch into the highest output throughput in the figure, at 79 tokens per second.

We kept llama.cpp’s default --parallel 4 because --parallel 16 improves throughput at concurrency 16 but regresses at concurrency 8 (sensitivity results).

Qwen3.6-35B-A3B

Qwen3.6-35B-A3B is a mixture-of-experts model: 35B total parameters with 3B active per token, and experts holding 93% of the checkpoint’s text weight. It also alternates standard attention with gated-delta-net linear attention, so one arm exercises both the hybrid attention path and the expert path.

SiliconBench agent split on Qwen3.6-35B-A3B: TTFT, end-to-end request latency, and output token throughput versus concurrency for llama.cpp, vllm-metal, oMLX with its prefix cache in memory and on SSD, and mlx_lm

vllm-metal holds TTFT nearly flat, 1.6 s to 2.0 s from concurrency 1 to 4, while oMLX climbs to 4.9 s in memory and 6.8 s with SSD offload, and llama.cpp sits between them. Throughput is closer here than on the dense models: 35.3 tokens per second for vllm-metal at concurrency 4 against 33.9 for oMLX in memory, 30.1 with offload, and 25.3 for llama.cpp, and end-to-end latency is within a tenth of a second (8.46 against 8.57). Single-stream, oMLX leads throughput outright, 33.7 against 28.3. On a model this sparse the scheduler’s advantage shows up in time to first token rather than in tokens per second. mlx_lm is drawn for reference but completed 38, 40 and 38 of 100 requests, so its line is not a like-for-like comparison.

Sparse expert routing does not change the shape of the result. The packed token axis carries through the MoE layers the same way it does through attention and MLP, so the padding a batch would otherwise spend is saved in all three.

Batched MTP under concurrent load

In vllm-metal, MTP drafting and verification stay inside the continuous-batching path. The dashed blue line in the Gemma 4 figure measures this path:

Concurrency Wall vs. no MTP Output tok/s vs. no MTP TTFT avg vs. no MTP
1 −15% +20% −1%
8 −1% +0% +4%
16 −8% +9% +20%

MTP pays most single-stream, adding a fifth to output throughput at concurrency 1, and still returns 9% at concurrency 16. At concurrency 8 the drafter’s cost exactly cancels the accepted tokens. The tradeoff is TTFT, which rises 20% at the top of the sweep. Drafting one token per step, the target model accepts 71% of drafts.

Draft depth does not extend the win. Running the drafter recurrently for more steps lowers acceptance — 71% at one draft token, 62% at two, 52% at three — but output throughput at concurrency 8 is flat across depth at 73.0, 73.7, and 72.6 tokens per second. The extra accepted tokens and the extra draft compute cancel. One draft token is the setting we recommend today, because it is the simplest and nothing deeper buys anything. Today the Metal MTP path is limited to Gemma 4 and requires --no-async-scheduling; the quickstart above includes both. MTP is opt-in through --speculative-config, so prefill-dominated deployments leave it off.

Other v0.28.0 additions

Faster prefill on M5

On M5 Macs, vllm-metal automatically uses the NAX kernel for compatible prefill batches; pre-M5 Macs keep using the existing path.

NAX prefill kernel A/B benchmark: TTFT, throughput, and TPOT with the tensor units on and off

NAX cuts mean TTFT by 41% on the prefill-heavy split and 26% on the standard split, while total throughput rises 33% and 8%. It also lowers TPOT by 25% and 7% because faster chunked prefill returns time to active decode streams.

Reusing conversation history on hybrid models

Multi-turn agents resend most of their growing conversation on every turn. Prefix caching lets the next turn reuse blocks computed for earlier turns instead of prefilling the full history again.

For Qwen3.5-style hybrid models, vllm-metal supports vLLM’s align mode. It checkpoints GDN recurrent state at the same block boundaries as attention KV, so both parts of the model can resume from the same cached prefix. A small custom Metal scatter kernel updates only the GDN state rows that changed, in place, without copying the whole shared state pool (PR #634).

On an M5 Pro repeated-prefix workload, this made Qwen3.5-0.8B finish the 100-request run about one-fifth sooner; unrelated prompts stayed within run-to-run noise. vLLM 0.28 enables align-mode prefix caching by default for supported hybrid models. The Metal path remains experimental and cannot yet be combined with speculative decoding.

Models and serving features

v0.28.0 also ships:

  • LoRA adapters, structured outputs, and three speculative-decoding methods: Gemma 4 MTP, separate draft models, and prompt-lookup n-grams.
  • GGUF checkpoints, including Hugging Face config sources for local GGUF weights.
  • Hybrid-attention models: the Qwen3.5, Qwen3.6, Qwen3.8, and Qwen3-Next families alternate standard attention with gated-delta-net linear attention, and Qwen3.6 adds mixture-of-experts on top; v0.28.0 serves mlx-community/Qwen3.8-27B-4bit and mlx-community/Qwen3.6-35B-A3B-4bit on a single Mac.
  • Pipeline parallelism across multiple Macs over the MLX ring backend.
  • Experimental vision-language models, text embeddings and reranking, and speech-to-text.

The supported-model matrix and feature guides are in the vllm-metal documentation.

The same stack from M1 Pro to M5 Pro

The benchmarks above run on an M5 Pro. To show what the same serving stack does on an older machine, we run the SiliconBench agent split on Gemma 4 E4B with an M1 Pro 32 GB alongside the M5 Pro 64 GB. The two machines differ in chip generation and memory, so the gap at concurrency 1 is a clean generational signal while higher concurrency also reflects the larger KV budget.

Concurrency TTFT avg (s)   Output throughput (tok/s)  
  M1 Pro 32 GB M5 Pro 64 GB M1 Pro 32 GB M5 Pro 64 GB
1 — 0.64 — 45.8
8 — 1.19 — 72.7

Appendix: benchmark reproduction

The cross-engine serving benchmarks use the SiliconBench agent split: 100 prompts averaging 4.6K input and 70 output tokens, run closed loop at fixed concurrency on a 64 GB M5 Pro. The NAX A/B instead uses two Sonnet configurations: a prefill-heavy split with 2,048 input and 32 output tokens, and a standard split with 1,024 input and 128 output tokens. Both run 100 prompts at request rate 10 and concurrency 32.

Each concurrency level runs against a freshly started server, so no level inherits a prefix cache warmed by the one before it; oMLX also gets an empty cache directory each time, because its cache is on disk. Its two arms differ only in which tier holds that cache. The SSD arm is oMLX’s own default — disk-backed, memory tier off, the 100 GB cap being what its auto setting resolves to on a 1 TB machine. The RAM arm holds the cache in memory and writes nothing to disk, matching what llama.cpp and vllm-metal do. The fresh directory is the one departure from a stock install in either arm, and it is what makes the levels independent. Each engine runs at its own memory default: vllm-metal at auto, which resolves to the engine’s --gpu-memory-utilization of 0.92; oMLX under its balanced memory guard; llama.cpp uncapped. The quickstart above deliberately recommends lower values so a Mac keeps headroom for other applications, and that costs throughput — on this model at concurrency 1, moving 0.5 to 0.7 to the default raises output from 3.9 to 4.4 to 4.7 tokens per second.

Stats cover completed requests; an empty response counts as failed. The benchmark code and per-engine configurations live in the SiliconBench repo. All arms ran on vllm-metal 0.28.0.dev20260901062632 against vLLM 0.28.0, llama.cpp 0eadefeb, and oMLX dc312e6e, with the serve commands from the quickstart. Every result file records the framework version that produced it.

Serving configurations 
# llama.cpp
llama-server -m <model>.gguf --host 0.0.0.0 --port 8001 \
  -ngl 99 --parallel 4 -c 65536

# vllm-metal
vllm serve <model> --host 0.0.0.0 --port 8004 \
  --enable-prefix-caching --max-model-len 16384

# vllm-metal + MTP (Gemma only)
vllm serve <model> --host 0.0.0.0 --port 8004 \
  --enable-prefix-caching --max-model-len 16384 --no-async-scheduling \
  --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

# oMLX, SSD offload — its own default apart from the fresh directory
omlx serve --model-dir <dir> --host 0.0.0.0 --port 8005 \
  --paged-ssd-cache-dir <fresh-empty-dir> --paged-ssd-cache-max-size 100GB \
  --hot-cache-max-size 0

# oMLX, RAM cache — same, but the cache stays in memory
OMLX_HOT_CACHE_ONLY=true omlx serve --model-dir <dir> --host 0.0.0.0 --port 8005 \
  --paged-ssd-cache-dir <fresh-empty-dir> --paged-ssd-cache-max-size 100GB \
  --hot-cache-max-size 8GB

llama.cpp’s context is divided across slots, giving 16,384 tokens per slot; the agent split’s longest prompt is 8.7K tokens. vllm-metal runs at its default memory fraction of auto, which resolves to a --gpu-memory-utilization of 0.92. oMLX’s cache directory must exist even in RAM mode — it builds no cache without one — but stays empty, and both oMLX arms get a fresh one before every concurrency level.

llama.cpp server-slot sensitivity

The main figures use llama.cpp’s default four server slots. A 16-slot sensitivity run improves output throughput at concurrency 16 but regresses at concurrency 8:

Split Concurrency --parallel 4 --parallel 16 Change
Chat 1 22.4 tok/s 24.2 tok/s +8%
Chat 8 77.0 tok/s 42.7 tok/s −45%
Chat 16 81.5 tok/s 104.7 tok/s +28%
Agent 1 18.3 tok/s 19.0 tok/s +4%
Agent 8 44.2 tok/s 26.7 tok/s −40%
Agent 16 40.2 tok/s 49.7 tok/s +24%

Acknowledgments

vllm-metal builds on MLX and mlx_lm from Apple’s MLX team, mlx-vlm for the vision-language paths, and vLLM’s engine and hardware-plugin interface. Thanks to the upstream vLLM maintainers for review and support along the way, and to everyone who filed issues and shared benchmarks against the v0.2 and v0.3 releases.