ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimizati | Hanami
ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concurrent agentic sessions. A B200 only has 180 GB of HBM, and after weight storage, only 100 GB of HBM per chip remains for KVCache.
THIS IS NOT ENOUGH FOR LONG-CONTEXT AGENTIC INFERENCE WORKLOADS. Thus, with SGLang HiCache, you can also offload KVCache to the CPU's DDR5/LPDDR5X memory, which helps improve performance. (1/2)🧵