https://vllm.ai/blog Technical articles, release announcements, model guides, and community updates from the vLLM project. en-us Tue, 03 Nov 2026 12:22:41 GMT https://vllm.ai/blog/2026-10-29-mistral-large-4 https://vllm.ai/blog/2026-10-29-mistral-large-4 Thu, 29 Oct 2026 00:00:00 GMT Day-0 vLLM support for Mistral Large 4 with sparse MLA, cross-layer KV sharing, DSpark speculative decoding, DCP/PCP long context and NIXL P/D on NVIDIA Blackwell. model-support speculative-decoding disaggregated-serving Mistral and the vLLM Team https://vllm.ai/blog/2026-10-23-deepseek-v41-flash-pd-wide-ep https://vllm.ai/blog/2026-10-23-deepseek-v41-flash-pd-wide-ep Fri, 23 Oct 2026 00:00:00 GMT With P/D disaggregation and Wide-EP on a GB300 NVL72 rack, vLLM serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU at a P90 of 196 tokens/s per user, 7.9x our aggregated single-node result. deepseek performance large-scale-serving disaggregation Inferact and the vLLM Team https://vllm.ai/blog/2026-10-23-glm53-part2-pcp-dcp-hisparse https://vllm.ai/blog/2026-10-23-glm53-part2-pcp-dcp-hisparse Fri, 23 Oct 2026 00:00:00 GMT vLLM serves GLM-5.3 on 16 H200s with PCP and DCP prefill and Hybrid HiSparse decode, reaching 1M-token contexts and 1.7–3.2× the per-user speed of data-parallel P/D on the AgentX agentic benchmark. glm parallelism kv-cache disaggregation performance vLLM Team https://vllm.ai/blog/2026-10-23-minimax-m31-flash https://vllm.ai/blog/2026-10-23-minimax-m31-flash Fri, 23 Oct 2026 00:00:00 GMT Day-0 vLLM support for MiniMax-M3.1-Flash on NVIDIA Blackwell, NVIDIA Rubin, and AMD MI350-series GPUs, with NVFP4 MSA kernels, NVFP4 MegaMoE, DSpark speculative decoding, AgentX results, and day-0 RL. minimax day-0-support performance kernels hardware Inferact and the vLLM Team https://vllm.ai/blog/2026-10-21-flint https://vllm.ai/blog/2026-10-21-flint Wed, 21 Oct 2026 00:00:00 GMT vLLM supports both Flint checkpoints (40B total, 2.5B active) from day 0, decoding a single request at about 500 tokens/s on one GPU. flint performance kernels Inferact and Recursive https://vllm.ai/blog/2026-10-21-humming https://vllm.ai/blog/2026-10-21-humming Wed, 21 Oct 2026 00:00:00 GMT Introducing Humming, a quantized inference kernel library from Ant Group and the vLLM community, with broad GPU and quantization support, dense and MoE performance results, and vLLM integration. quantization performance kernels moe Ant Group and the vLLM Community https://vllm.ai/blog/2026-10-20-vllm-torchtpu https://vllm.ai/blog/2026-10-20-vllm-torchtpu Tue, 20 Oct 2026 00:00:00 GMT vllm-torchtpu is a new vLLM backend for Google Cloud TPUs built on TorchTPU, with Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving. hardware Inferact and Google Teams https://vllm.ai/blog/2026-10-19-elastic-ep-near-zero-downtime https://vllm.ai/blog/2026-10-19-elastic-ep-near-zero-downtime Mon, 19 Oct 2026 00:00:00 GMT How asynchronous preparation and NIXL EP enabled near-zero-downtime Elastic EP scaling in vLLM. large-scale-serving elastic-ep expert-parallelism moe nixl cuda-graphs Itay Alroy (NVIDIA) https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel Fri, 16 Oct 2026 00:00:00 GMT How agent-assisted CuTe DSL optimizations accelerate Kimi K3 decode on NVIDIA GB300, and how scoped changes, correctness tests, and reproducible benchmarks help ship them responsibly. performance hardware NVIDIA Benjamin Chislett (NVIDIA) https://vllm.ai/blog/2026-10-16-kimi-k3-mi355x https://vllm.ai/blog/2026-10-16-kimi-k3-mi355x Fri, 16 Oct 2026 00:00:00 GMT Inside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster models performance rocm amd speculative-decoding AMD and Embedded LLM https://vllm.ai/blog/2026-10-16-reflection-beam https://vllm.ai/blog/2026-10-16-reflection-beam Fri, 16 Oct 2026 00:00:00 GMT vLLM supports Reflection's Beam-1.0 (501B total, 23B active) from day 0, with agent-driven optimization that cut decode latency 2.71× and DFlash2 speculative decoding with adaptive verification. reflection performance kernels speculative-decoding Inferact, Reflection, and Nvidia https://vllm.ai/blog/2026-10-15-muse-spark-open https://vllm.ai/blog/2026-10-15-muse-spark-open Thu, 15 Oct 2026 00:00:00 GMT Day-0 vLLM support for Muse Spark Open: 816B MoE, 1M context, multimodal inputs, DSpark speculative decoding, NVIDIA and AMD serving, initial TPU support, and LoRA RL. model-support speculative-decoding reinforcement-learning vLLM Team https://vllm.ai/blog/2026-10-15-nemotron-3-5-super-vl-vllm https://vllm.ai/blog/2026-10-15-nemotron-3-5-super-vl-vllm Thu, 15 Oct 2026 00:00:00 GMT Run NVIDIA Nemotron 3.5 Super VL on vLLM for multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction. model-support NVIDIA Nemotron Team https://vllm.ai/blog/2026-10-14-adaptive-verification-online-estimator https://vllm.ai/blog/2026-10-14-adaptive-verification-online-estimator Wed, 14 Oct 2026 00:00:00 GMT An online-fitted acceptance estimator that lets every draft-model speculator use Adaptive Verification, by predicting the target's acceptance probability from the drafter's own logits and refitting continuously during serving. speculative-decoding performance Giancarlo Delfin (Inferact) https://vllm.ai/blog/2026-10-14-nixl-push-connector https://vllm.ai/blog/2026-10-14-nixl-push-connector Wed, 14 Oct 2026 00:00:00 GMT A push-based NIXL connector for vLLM disaggregated serving: the prefill instance writes KV directly into decode memory over RDMA, taking the router relay, decode-side allocation, and transfer setup off the TTFT critical path. disaggregation kv_cache nixl performance Sunita Nadampalli (AWS), Nicolò Lucchesi (Mistral) https://vllm.ai/blog/2026-10-14-train-inference-consistency https://vllm.ai/blog/2026-10-14-train-inference-consistency Wed, 14 Oct 2026 00:00:00 GMT We make vLLM rollout and the training forward give bitwise identical logprobs on DeepSeek-V4-Flash and Nemotron-3.5-Lightning-30B-A3B (NVFP4), keep 91% of rollout throughput on DeepSeek-V4-Flash, and open-source the agent skill. reinforcement-learning deepseek Yan Bai (NVIDIA DevTech), Ao Shen (Inferact) https://vllm.ai/blog/2026-10-12-bidirectional-kvxfer-multiturn-agentic-workload https://vllm.ai/blog/2026-10-12-bidirectional-kvxfer-multiturn-agentic-workload Mon, 12 Oct 2026 00:00:00 GMT Bi-directional KV transfer lets vLLM prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts. disaggregation kv_cache nixl performance Sunita Nadampalli (AWS) https://vllm.ai/blog/2026-10-12-magi2-preview-musa https://vllm.ai/blog/2026-10-12-magi2-preview-musa Mon, 12 Oct 2026 00:00:00 GMT vLLM-Omni runs Sand AI's MAGI-2 Preview audio-video generation model on eight Moore Threads MTT S5000 GPUs through torchada's CUDA compatibility layer, then optimizes MoE, mHC, attention, parallelism, and compilation. performance hardware multimodal vllm-omni moe Moore Threads Team and vLLM-Omni Team https://vllm.ai/blog/2026-10-09-vera-rubin-preview https://vllm.ai/blog/2026-10-09-vera-rubin-preview Fri, 09 Oct 2026 00:00:00 GMT vLLM now runs on NVIDIA Vera Rubin NVL72 with day-0 model support, Rubin-tuned FlashInfer kernels, and locality-aware MoE, delivering 7.8x the per-GPU throughput of GB200 NVL72 on AgentX. hardware performance vLLM Team, Inferact, Red Hat, and NVIDIA https://vllm.ai/blog/2026-10-07-deepseek-v41-flash https://vllm.ai/blog/2026-10-07-deepseek-v41-flash Wed, 07 Oct 2026 00:00:00 GMT Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions. deepseek performance kernels Inferact and the vLLM Team https://vllm.ai/blog/2026-09-29-disaggregated-serving-guide https://vllm.ai/blog/2026-09-29-disaggregated-serving-guide Tue, 29 Sep 2026 00:00:00 GMT What disaggregated serving actually buys you, how to run it end to end in vLLM today with prefill/decode plus the new GPU-less frontend and the things we're still working on. disaggregation Martin Hickey (IBM Research) https://vllm.ai/blog/2026-09-24-watermarking-in-vllm https://vllm.ai/blog/2026-09-24-watermarking-in-vllm Thu, 24 Sep 2026 00:00:00 GMT How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-context safeguards. watermarking sampling speculative-decoding performance Raphaël Rialland (Mistral), Simon Veitner (Red Hat), and Tomas Ruiz (Red Hat) https://vllm.ai/blog/2026-09-22-python-bytecode-startup https://vllm.ai/blog/2026-09-22-python-bytecode-startup Tue, 22 Sep 2026 00:00:00 GMT Shipping Python bytecode saved about 18 seconds per startup in H100 server benchmarks, with images and models already local. performance deployment Nils Matteson https://vllm.ai/blog/2026-09-22-vllm-metal-v0-28-0 https://vllm.ai/blog/2026-09-22-vllm-metal-v0-28-0 Tue, 22 Sep 2026 00:00:00 GMT vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration. apple-silicon hardware performance ecosystem Ranran Haoran Zhang, Lik Xun Yuan, Chao Ju Chen, Eric Curtin, Michael Goin https://vllm.ai/blog/2026-09-21-qwen38-pd-serving https://vllm.ai/blog/2026-09-21-qwen38-pd-serving Mon, 21 Sep 2026 00:00:00 GMT How vLLM reaches 5K throughput and 180 interactivity on Qwen3.8-2.4T with GB300 NVL72 PD serving and how to reproduce results yourself. performance qwen3.8 disaggregation vLLM Team https://vllm.ai/blog/2026-09-18-pynvvideocodec https://vllm.ai/blog/2026-09-18-pynvvideocodec Fri, 18 Sep 2026 00:00:00 GMT How to leverage NVIDIA Hardware Video Decoders to Achieve Multi-GPU Scaling in Video Captioning and Description tasks. performance multimodal hardware NVIDIA Computer Vision Team (NVCV) https://vllm.ai/blog/2026-09-15-kimi-k3-dspark https://vllm.ai/blog/2026-09-15-kimi-k3-dspark Tue, 15 Sep 2026 00:00:00 GMT How Speculators and Mooncake enabled multi-node DSpark training for Kimi K3. speculators speculative-decoding performance distributed Helen Zhao, Fynn Schmitt-Ulms, Yuchen Fama, Antonio J. Dominguez, and Kevin Li https://vllm.ai/blog/2026-09-15-novita-chord-w4a16-moe https://vllm.ai/blog/2026-09-15-novita-chord-w4a16-moe Tue, 15 Sep 2026 00:00:00 GMT Novita AI has open-sourced Chord, a high-performance W4A16 MoE CUDA kernel for Kimi K2.x serving shapes, with a Humming-compatible indexed path and grouped SM90 operators. performance quantization moe hardware Novita AI and the vLLM Team https://vllm.ai/blog/2026-09-14-rl-kernel-v0-1-0-train-rollout-bitwise-consistency https://vllm.ai/blog/2026-09-14-rl-kernel-v0-1-0-train-rollout-bitwise-consistency Mon, 14 Sep 2026 00:00:00 GMT vime and RL-Kernel align selected-token logprobs bit for bit across Megatron training and vLLM rollout on AMD Instinct MI300X, with zero mismatches across 200 GRPO steps. reinforcement-learning post-training performance hardware ecosystem RL-Kernel Team, vime Team, and AMD Team https://vllm.ai/blog/2026-09-13-kimi-k3-performance-optimization https://vllm.ai/blog/2026-09-13-kimi-k3-performance-optimization Sun, 13 Sep 2026 00:00:00 GMT Kimi K3 serving optimizations across scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels. models performance kernels speculative-decoding Wentao Ye (Red Hat), Canlin Guo (Huawei), Yongye Zhu (Inferact), Jiangyun Zhu (Inferact), Ziming Huang (Inferact), Wei Zhao (NVIDIA), Michael Goin (Red Hat), Jie Li (Inferact) https://vllm.ai/blog/2026-09-10-minimax-m3-mi355x https://vllm.ai/blog/2026-09-10-minimax-m3-mi355x Thu, 10 Sep 2026 00:00:00 GMT A performance model for LLM serving: inspect local shapes, remove repeated work, verify data movement and dispatch, then follow the queue. hardware performance moe speculative-decoding disaggregation AMD and Embedded LLM Teams https://vllm.ai/blog/2026-09-10-tiered-kv-offloading https://vllm.ai/blog/2026-09-10-tiered-kv-offloading Thu, 10 Sep 2026 00:00:00 GMT A host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers — reducing recomputation and increasing serving capacity. Or Ozeri, Danny Harnik, Ronen Schaffer, Itay Etelis, Varun Sundar Rabindranath https://vllm.ai/blog/2026-09-08-glm53-part1-hybrid-sparse-offloading https://vllm.ai/blog/2026-09-08-glm53-part1-hybrid-sparse-offloading Tue, 08 Sep 2026 00:00:00 GMT vLLM integrates HiSparse as a pressure-driven memory tier that composes with the Hybrid Memory Allocator and KV offloading, letting GLM 5.3 requests keep decoding when their KV no longer fits in GPU memory, so concurrency stays high. glm kv-cache performance vLLM Team https://vllm.ai/blog/2026-09-08-vllm-agentx https://vllm.ai/blog/2026-09-08-vllm-agentx Tue, 08 Sep 2026 00:00:00 GMT How vLLM optimizes KV cache management, parallelism, scheduling, and P/D disaggregation for agentic workloads, validated on SemiAnalysis AgentX with up to 130K tokens per GPU-second and a 14.6x-106x serving-cost advantage over Opus 5. agentic kv_cache parallelism large-scale-serving disaggregation performance vLLM Team and Inferact https://vllm.ai/blog/2026-09-07-vllm-tt-plugin https://vllm.ai/blog/2026-09-07-vllm-tt-plugin Mon, 07 Sep 2026 00:00:00 GMT Tenstorrent accelerators join vLLM as an out-of-tree platform plugin, driven by mesh-architecture choices: phase-based scheduling, single-process data parallelism on Galaxy, on-device sampling with host fallback, and async decode overlap. hardware ecosystem Tenstorrent Team https://vllm.ai/blog/2026-09-01-minimax-h3-production-serving https://vllm.ai/blog/2026-09-01-minimax-h3-production-serving Tue, 01 Sep 2026 00:00:00 GMT How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback. performance large-scale-serving multimodal vllm-omni fastvideo fasth3 vLLM-Omni Team https://vllm.ai/blog/2026-08-23-speculative-decoding-amd-gpus https://vllm.ai/blog/2026-08-23-speculative-decoding-amd-gpus Sun, 23 Aug 2026 00:00:00 GMT A practical guide to speculative decoding in vLLM on AMD GPUs, covering draft-and-verify mechanics, MTP, EAGLE-3, DFlash, DSpark, configuration, tuning, and benchmark results. speculative-decoding amd AMD and Embedded LLM https://vllm.ai/blog/2026-08-22-rdt-weight-transfer https://vllm.ai/blog/2026-08-22-rdt-weight-transfer Sat, 22 Aug 2026 00:00:00 GMT We implement a native sharded weight transfer engine in vLLM utilizing Ray Direct Transport (RDT), achieving weight transfer for the Kimi K2 model in BF16 on 48 8xH100 nodes in 7.53s reinforcement-learning performance Aaron Hao, Sumanth Hegde, Gal Meirom, Istvan Haller, Kourosh Hakhamaneshi, Gavin Parnaby, Moein Khazraee, Omri Kahalon https://vllm.ai/blog/2026-08-21-isoexec https://vllm.ai/blog/2026-08-21-isoexec Fri, 21 Aug 2026 00:00:00 GMT IsoExec unifies numerical execution across SkyRL's vLLM and Megatron runtimes, reducing the average rollout-versus-training logprob difference below 1e-6 on Qwen3.5-35B-A3B with 25% overhead. reinforcement-learning performance Alexander Jiang and the SkyRL Team https://vllm.ai/blog/2026-08-20-verl-omni-v0-2-0 https://vllm.ai/blog/2026-08-20-verl-omni-v0-2-0 Thu, 20 Aug 2026 00:00:00 GMT A release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage. multimodal rlhf ecosystem performance VeRL-Omni Team https://vllm.ai/blog/2026-08-17-distributed-layerwise-offload https://vllm.ai/blog/2026-08-17-distributed-layerwise-offload Mon, 17 Aug 2026 00:00:00 GMT Distributed Layerwise Offload shards and streams DiT weights across devices, serving a measured 124 GB Cosmos3 model on 64 GB HBM and estimating a path toward 200B+ models. performance distributed vllm-omni cosmos3 vLLM-Omni Diffusion Team https://vllm.ai/blog/2026-08-14-dspark-adaptive-verification https://vllm.ai/blog/2026-08-14-dspark-adaptive-verification Fri, 14 Aug 2026 00:00:00 GMT Sizing the DSpark draft-verification budget from per-request confidence instead of verifying every drafted token, so one configuration holds the throughput/latency frontier from batch size 1 to 256. performance speculative-decoding vLLM Team https://vllm.ai/blog/2026-08-12-qwen3.8 https://vllm.ai/blog/2026-08-12-qwen3.8 Wed, 12 Aug 2026 00:00:00 GMT Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware. model-support quantization moe performance hardware vLLM Team and Inferact https://vllm.ai/blog/2026-08-10-nemotron-3-5-lightning-vllm https://vllm.ai/blog/2026-08-10-nemotron-3-5-lightning-vllm Mon, 10 Aug 2026 00:00:00 GMT How vLLM serves NVIDIA Nemotron 3.5 Lightning with OpenAI-compatible APIs, speculative decoding, and BF16/NVFP4 checkpoints across NVIDIA GPUs and edge systems. model-support NVIDIA Nemotron Team and vLLM Team https://vllm.ai/blog/2026-08-07-decode-context-parallelism https://vllm.ai/blog/2026-08-07-decode-context-parallelism Fri, 07 Aug 2026 00:00:00 GMT Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads compared to standard tensor parallelism. performance attention parallelism Seonghee Lee, Sungsoo Ha, Omri Almog (NVIDIA), Lucas Wilkinson (Red Hat AI) https://vllm.ai/blog/2026-08-06-qwen35-25k-tps https://vllm.ai/blog/2026-08-06-qwen35-25k-tps Thu, 06 Aug 2026 00:00:00 GMT How vLLM reaches 25K total TPS/GPU on Qwen3.5-397B-A17B-NVFP4 with GB200 NVL72 disaggregated serving, Blackwell GDN kernels, HMA cache transfer, async scheduling fixes, and srt-slurm recipes. performance qwen3.5 disaggregation vLLM Team https://vllm.ai/blog/2026-07-29-optimizing-vllm-on-arm-cpus https://vllm.ai/blog/2026-07-29-optimizing-vllm-on-arm-cpus Wed, 29 Jul 2026 00:00:00 GMT An overview of Arm CPU enablement and inference performance optimizations in vLLM. hardware performance Arm Team https://vllm.ai/blog/2026-07-28-speculators-parallel-drafting https://vllm.ai/blog/2026-07-28-speculators-parallel-drafting Tue, 28 Jul 2026 00:00:00 GMT Speculators and vLLM now support P-EAGLE, DFlash, and DSpark — three parallel drafting algorithms that move beyond sequential token generation to deliver faster, simpler, and more scalable speculative decoding for LLM serving. speculators speculative_decoding peagle dflash dspark Alexandre Marques, Megan Flynn, Helen Zhao, Krishna Teja Chitty Venkata, Chibueze Ukachi (Red Hat AI) https://vllm.ai/blog/2026-07-27-k3 https://vllm.ai/blog/2026-07-27-k3 Mon, 27 Jul 2026 00:00:00 GMT vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across NVIDIA and AMD GPUs. models performance prefix caching multimodal vLLM Team and Inferact https://vllm.ai/blog/2026-07-23-glm-5.2-nvfp4-b300-pd https://vllm.ai/blog/2026-07-23-glm-5.2-nvfp4-b300-pd Thu, 23 Jul 2026 00:00:00 GMT How we took GLM-5.2-NVFP4 from 40 ms to 17 ms mean TPOT on 24 B300 GPUs with vLLM: P/D disaggregation, MTP speculative decoding, Model Runner V2, and the SLA-first trade-offs behind the final configuration. disaggregation performance speculative-decoding moe large-scale-serving DaoCloud Team