Cloudflare Optimizes LLM Inference with KV Cache Quantization and Weight Compression
Cloudflare details significant performance and memory optimizations for running large language models Kimi and GLM on its Workers AI platform, employing techniques like FP8 KV cache quantization and INT4 weight compression.
Cloudflare is pushing the boundaries of efficient large language model (LLM) deployment on its Workers AI platform, detailing innovative techniques to run demanding models like Moonshot's Kimi K-series and Z.ai's GLM with greater speed and reduced memory footprint. These models, known for their long context capabilities and mixture-of-experts architecture, present substantial serving challenges due to their size and memory requirements.
The company's approach builds upon previous strategies, including optimizing inference by separating prefill and decode phases. This latest advancement focuses on three key optimizations: quantizing the KV cache, compressing model weights, and implementing integrity checks for the shared KV cache. These methods collectively aim to enhance memory efficiency and throughput, enabling higher concurrency and lower operational costs without compromising model accuracy.
A primary focus of Cloudflare's optimization efforts is the KV cache, which stores attention keys and values generated during text creation. For long-context models, this cache can grow rapidly and often becomes the bottleneck in GPU memory. By quantizing the KV cache from its default 16-bit floating point (BF16) precision to 8-bit floating point (FP8), Cloudflare effectively halves its size. This doubles the context capacity, allowing models like Kimi K2.6 to handle approximately 1.37 million tokens, up from 686,000, on the same hardware.
While FP8 quantization slightly increases the computational work per token due to necessary value conversions, its primary benefit lies in enabling more requests to reside in memory simultaneously. Benchmarks show that FP8 can support up to 64 concurrent requests, compared to BF16's limit of 32, leading to a significant increase in throughput and a reduction in cost per token. Cloudflare strategically applies FP8 to the decode phase, where memory is the constraint, while retaining BF16 for the compute-bound prefill phase to leverage its slightly higher raw speed.
In addition to KV cache optimization, Cloudflare has implemented weight compression for models like GLM 5.2. By compressing model weights from 8-bit floating point to 4-bit integers (INT4), the checkpoint size is reduced by approximately 40%, from 705 GB to 421 GB. This dramatically lowers per-GPU memory requirements, freeing up space for a larger KV cache and enabling models to handle over a million tokens of context. This compression also speeds up the decode phase by reducing the amount of data that needs to be streamed from memory.
Similar to the KV cache strategy, Cloudflare applies INT4 compression selectively. It is used for the decode phase, where it offers a direct speed advantage due to reduced memory bandwidth usage. For the compute-bound prefill phase, where INT4 weights require an expansion step that slows down processing, the company opts for the uncompressed FP8 weights. Crucially, extensive evaluations across various benchmarks confirm that both FP8 KV cache quantization and INT4 weight compression maintain model accuracy, with performance differences remaining within negligible margins.
To safeguard the integrity of the shared KV cache, especially with hundreds of requests concurrently accessing it, Cloudflare has developed a KV cache integrity checking mechanism. This system uses tags to track physical cache pages and verifies mappings before decode operations, aborting requests that show inconsistencies. This defense layer adds minimal overhead, less than 1% to throughput and latency, ensuring the reliability of the optimized inference process.
These advancements, developed in conjunction with the open-source SGLang inference serving framework, underscore Cloudflare's commitment to making powerful AI models more accessible and cost-effective. By pushing the envelope in model optimization, Cloudflare aims to empower developers with high-performance AI capabilities directly at the network edge.