vLLM — the open-source inference engine that became the default way to serve large language models — shipped version 0.28.0 in August 2026, and it is one of the project’s meatiest releases. The numbers: 584 commits from 270 contributors, 76 of them first-timers. The substance: a full-stack performance push for Kimi-K3, end-to-end support for DeepSeek V4’s sparse MLA, speculative decoding advances, a maturing Model Runner V2, tiered KV-cache offloading to disk, and a new Rust frontend with gRPC.
If you serve open-weights models in production, this release touches your stack. Here is what landed.
The Kimi-K3 performance push
The single largest effort in 0.28 targets Moonshot’s Kimi-K3, the 2.8-trillion-parameter MoE that reset expectations for open models in July. The optimizations run the full stack: Decode Context Parallel (DCP) support, fused FlashKDA decode and prefill kernels, SiTU activation support for the MegaMoE architecture, GEMM-RS for sequence parallelism, and combined all-gathers delivering 1.5–3x kernel-level speedups.
Two items stand out. An adaptive speculative token budget delivers roughly 60% better DSpark time-to-first-token, and optional shared-expert sharding saves about 17 GiB of memory per GPU — real money at K3’s scale. Kimi-K3 also now runs on ROCm via the V2 model runner, widening the hardware story beyond NVIDIA.
DeepSeek V4 and speculative decoding
DeepSeek’s V4 architecture gets end-to-end sparse MLA support across plain decode, multi-token prediction, and DSpark speculative decoding, plus AMD Quark NVFP4 quantization support, reasoning-effort prompt mappings, sparse top-k metadata kernel optimizations, and ROCm enablement on gfx11 and gfx950.
Speculative decoding — the technique of using a small draft model to propose tokens the big model verifies — advances on three fronts: DFlash2 with local convolution and a candidate selector, DSpark confidence-scheduled verification, and async scheduling auto-enabled for draft models. For latency-sensitive serving, this is where the release’s real-world wins live.
Model Runner V2 matures, KV cache goes tiered
The V2 model runner, vLLM’s ongoing architectural overhaul, gains encode/prefill/decode disaggregation — splitting the inference pipeline across specialized hardware — plus weight offloading, multi-layer MTP KV cache support, encoder CUDA graphs, decoder token-wise pooling, attention-free model support, and thinking-token budgets for reasoning models.
Alongside it comes tiered KV-cache offloading: disk offloading support, out-of-tree secondary-tier managers loadable via module path, partial secondary-tier load results, tiering metrics, and a canonical CPU layout for parallelism-agnostic offload. In plain terms: vLLM can now spill KV cache to slower storage in a structured way, which changes the economics of serving long-context workloads without renting more HBM than you need.
The Apple Silicon side quest
While upstream vLLM chases datacenter throughput, its Apple Silicon plugin — vllm-metal — is having its own moment. The plugin brings vLLM’s scheduler, paged KV cache, and OpenAI-compatible server to Macs via MLX and Metal, with its v0.28.0 release aligning version numbering with upstream. August brought M5 NAX tensor-unit acceleration for attention prefill, and Qwen3.8-27B now serves on a single Apple Silicon Mac. It is a reminder that the vLLM ecosystem spans both ends of the hardware spectrum: thousand-GPU clusters and a MacBook.
Rust frontend, gRPC, and new defaults
A new Rust frontend and gRPC interface bring a standalone renderer, multimodal image inference over gRPC, explicit data-parallel rank routing, and RL lifecycle control, with protobuf schemas published to Buf. For teams building non-Python serving paths or RL training loops on vLLM, this is a meaningful expansion of the project’s surface area.
New defaults worth knowing: max_num_batched_tokens rises from 8192 to 16384, prefix caching is now on by default for Mamba models, and the Blackwell CUDA graph capture default moves to 1024. And the breaking changes: bitsandbytes support migrates to an out-of-tree plugin, Transformers bumps to 5.15.0, and the deprecated calculate_kv_scales runtime KV-scale calculation and override_attention_dtype are removed. Check your configs before upgrading.
The inference wars
vLLM no longer has the field to itself. SGLang — built around RadixAttention, a radix-tree KV-cache manager that reuses shared prompt prefixes — is now deployed across more than 400,000 production GPUs at xAI, AMD, NVIDIA, Intel, LinkedIn, and Cursor, according to September 2026 analysis. On prefix-heavy workloads like multi-turn agent conversations, independent benchmarks show SGLang roughly 29% ahead on throughput with 4.5x faster time-to-first-token; on unique-prompt traffic the two tie, and vLLM’s broader hardware support keeps it the safer default.
The 0.28 release reads as vLLM’s answer: disaggregated prefill/decode, tiered KV offloading, and the Kimi-K3 optimization blitz are all aimed at the exact workloads where SGLang has been gaining. Competition between two excellent open-source engines is the best possible outcome for operators — and this release keeps vLLM firmly in the fight.
Getting it
Wheels ship for CUDA 13.0 (pip install vllm), ROCm, CPU, and XPU, with Docker images for each platform. For a hands-on setup walkthrough, see our guide to serving open models with vLLM. vLLM remains Apache 2.0 licensed and PyTorch Foundation-hosted — the boring, important facts that let enterprises build on it.
The timing is no accident: 0.28 is the release that makes serving this summer’s giant open models — Ornith-1.5’s 397B flagship, Qwen3.8-Max, Kimi-K3 — practical rather than theoretical. The weights are free; the serving is the product. And with 270 contributors landing 584 commits in a single release cycle, vLLM’s development velocity remains the project’s quiet superpower — no closed serving stack is improving this fast in public. For operators, the upgrade checklist is short: review the breaking changes, test the new batching defaults under your own traffic shape, and benchmark prefix-heavy workloads before deciding where SGLang fits. Either way, the era of serving giant open models on a single default engine is over — and that is good news.
Featured image: Wil Weterings via Wikimedia Commons (Public domain)
