Highlights Model Support Support Qwen3.5 checkpoints with global FP8 scales across model loading paths #19519 Optimize Gemma4 vision rotary embeddings without a GEMM operation #19571 Enable the CuTeDSL MoE backend for Kimi K3 NVFP4 SiTU #19003 Expose Nemotron-H vision-language LoRA configuration for supported inference paths #19151 Extend MiniMax-M3 piecewise CUDA graphs through model and executor…
Highlights Known Issues Multi-node inference may fail during startup when a requested parallel mapping spans nodes or request payloads reference caller-local Python modules. Disaggregated serving may fail to start when an advertised hostname does not resolve to a bindable IPv4 address. Use a hostname that resolves to a local IPv4 interface. Qwen3.5-4B may lose accuracy with context pipeline parall…
Highlights Known Issues GPT-OSS Eagle3 speculative decoding may hang, produce non-terminating output, or lose accuracy when using multi-GPU execution. GPT-OSS W4 inference with the Triton MoE backend may intermittently hang or lose accuracy in four-GPU data-parallel or expert-parallel configurations. GPT-OSS Harmony tool calls may continue generating after the tool-call delimiter, resulting in mal…
Highlights Known Issues LoRA adapters that target the routed experts of Qwen3 mixture-of-experts models are silently ignored and produce base-model output. Adapters targeting attention layers behave correctly. Models using the V2 KV cache manager may fail to start with an insufficient-memory error even when enough GPU memory is free, because the KV cache size estimate is too conservative. GPT-OSS…
Highlights Annoucements KV Cache Manager Enabled KV cache manager V2 by default for: DeepSeek V3, DeepSeek R1, and DeepSeek V3 Lite DeepSeek V3.2 DeepSeek V4 GLM-5 GPT-OSS Mistral Large 3 Kimi K2, Kimi K2.5, and Kimi K3 MiniMax M2 and MiniMax M3 Nemotron H and Nemotron H Puzzle Qwen3-Next, Qwen3.5, and Qwen3.8 Gemma 3 and Gemma 4 KV cache manager V2 is the recommended architecture, with improved s…
Signed-off-by: Yanchao Lu yanchaol@nvidia.com
Signed-off-by: Yanchao Lu yanchaol@nvidia.com
Known Issues Workloads combining torch.compile with CUDA graphs can fail during initialization or inference with allocator assertions, missing outputs, or illegal memory accesses. MLA models using MTP with multi-stream or piecewise CUDA graphs on SM120 can asynchronously crash with an illegal memory access. Workloads using KVCacheManagerV2 may intermittently report a teardown failure after otherwi…
Known Issues Deepseek-V4-Pro can hang on GB300 disagg setups. DeepSeek-R1 NVFP4 multi-GPU (PP4 + MTP) can crash with an MPI worker exit during execution (seen on GB300). DeepSeek-V3-Lite BF16 + chunked prefill (Python scheduler) can hang / fail to complete. Qwen MoE multi-LoRA (routed experts, varying ranks, eager) can fail during sampling. Gemma3-1B FP8 prequantized with torch.compile can fail du…
Known Issues torch.compile crashes in the PyTorch compilation backend Several multi-GPU accuracy paths fail with KeyError in the remove_copy pass when torch_compile=True (e.g. DeepSeek-V3-Lite bf16/FP8/NVFP4; Llama-3.1-8B-Instruct FP8 with FlashInfer). Workaround: disable torch.compile for affected configs DeepSeek-V3.2 FP8 block-scale OOM on H200. test_fp8_blockscale configs can OOM on DGX H200 u…
Deprecation Notices AutoDeploy backend is being deprecated. We are aware that earlier model support is a critical priority for many users of TensorRT LLM and are working on agentic approaches to improve time to functional model support in the PyTorch backend. As an early indicator, this was used to release Minimax M3 functional support within the first week of model release. Known Issues DeepSeek…
This RC version will be the last one supporting the TensorRT backend, in the next version the TensorRT backend will be removed! Known Issues DeepSeek V3/V3.2 can crash with an illegal memory access or hang during warm up. Autotuning for Qwen3-family models can crash with "Assertion failed: Failed to initialize cutlass TMA WS grouped gemm." API Add API to configure TeaCache coefficients (#13170) BR…
Known Issues Llama 3.1 8B FP8 can hang during the autotuner warmup on GB200. Model Support Support NVIDIA Wan2.2-T2V quantized checkpoints (#15093) Enable MTP for Step-3.7 NVFP4 and port Step-3.7VL vision tower to TRT-LLM modules (#14926) Support T5 and BART in the PyTorch backend (#13919) Support MiniMax-M3 in the PyTorch backend (#15292) API Align VisualGen serve request schema with VisualGenPar…
Known Issues DSV3.2 will crash with an IMA in various long-running perf tests on GB200/GB300 when the CuteDSL MoE backend is used. Work around this issue by using another MoE backend. Model Support Support Nemotron-H NVFP4 checkpoint on Hopper (#14775) Add Qwen image support (#13449) Support Step-3.7-Flash model (#14711) Add Cosmos3-Nano and Cosmos3-Super support (#14824) Add AFMoE Trinity support…
Highlights Known Issues DeepSeek V3.2 will crash with an illegal memory access during long-running performance tests under various agg/disagg configurations. Model Support Add MoT World Model support (#14012) Enable multi-node tensor parallelism for MiniMax-M2 (#14314) Restore Mistral Large 3 text-only processor (#14248) Support Gemma4 multi-head_dim pools and host-side slicing for SWA Triton kern…
Signed-off-by: Yiteng Niu 6831097+niukuo@users.noreply.github.com
Highlights Model Support Add Gemma4 multimodal support with native vision and audio towers (#14300) Add Qwen3.5 MTP and Qwen3.6-27B-FP8 model support (#12646, #14359) Add EXAONE-4.5 and Laguna model support (#12873, #13559) Switch DeepSeek, NemotronH, Qwen3, and Qwen3.5-MoE to sharding-IR canonical models (#13478) API Refactor the VisualGenArgs API and registry (#14175) Drop sink_token_length from…
Highlights Model Support Add Gemma4 multimodal model support with text, vision, audio, and chunked prefill capabilities (#12932, #14134) Add Kimi K2.5 multimodal vision support and reasoning parser integration (#12788, #13801) Add GPT-OSS, Ministral3, Nemotron-H, Nemotron Nano, and DeepSeek model enablement and compatibility updates (#12743, #12884, #13844, #13977) Improve DeepSeek V4 and DeepSeek…
Highlights Model Support Add prefix caching for Mamba hybrid models including Qwen3.5 and Nemotron Super V3 (#12185) Improve Qwen3.5 support with custom MoE routing and dense and NVFP4 weight loading fixes (#13433, #13090, #13716) Improve Nemotron and Nemotron Nano support with GEMM tuning and multimodal placeholder expansion (#13160, #13069) Add Wan 2.2 5B TI2V support and refine LTX-2 FP4 stage…
Highlights Model Support Support and initial optimizations for Nemotron 3 Nano Omni; known issues for audio-from-video and chunked prefill for video being actively worked on Add audio extraction from video, optimize ViT attention, and reduce initialization memory for Nemotron and Nemotron Nano VL models (#12921, #12911, #13283) Add per-model VisualGen example scripts, shared configs, per-model def…
[None][feat] Add conversation router integration test and improve fin…