This release introduces RL serving with verl, disaggregated ViT serving, a hybrid-cache architecture for linear-attention models, and a multi-level cache system with cache-aware PD. It also delivers broad model-performance improvements—including expanded MTP speculative decoding and optimizations across prefill, attention, MoE, quantization, and distributed inference—together with extensive stabil…
🎉 Announcing LightLLM v1.1.0: More Efficient, More Powerful! We are thrilled to introduce LightLLM v1.1.0, featuring major architectural and optimization upgrades for higher performance and broader applicability. ✨ Key Highlights 🚀 CPU-GPU Unified Folding Architecture Drastically reduces system-level CPU overhead ⚡ Deep Model Optimizations Enhanced support for DeepSeek and Qwen3-MoE Integration of…
Highlights DeepSeek-R1 Multi-Node H100 Deployment Support FlashInfer Integration XGrammer Integration What's Changed Benchclient by @shihaobai in #740 fix pause reqs by @shihaobai in #741 add RETURN_LIST for tgi_api by @shihaobai in #742 fix: fix a precision bug in the context_flashattention by @blueswhen in #743 Improve the accuracy of deepseekv3 by @hiworldwzj in #744 deepseekv3 bmm noquant and…
New Features Cross-Process Request Object: Retained and optimized the previous three-process architecture design. Introduced a request object that can be accessed across processes, significantly reducing inter-process communication overhead. Folding of scheduling and model inference: Implemented the folding of scheduling and model inference, significantly reducing communication overhead between th…