Cloudflare Discloses Tenant Isolation Flaw in GLM-5.3 Workers AI Model

Cloudflare has disclosed a significant tenant isolation vulnerability that affected its GLM-5.3 Workers AI model, impacting a subset of customer responses between August 27 and September 9, 2026. The issue, reported by security researcher Gadi Evron of Knostic, allowed responses to inadvertently include data from other customers' sessions. Cloudflare has since remediated the vulnerability and restored the model, confirming that no other models hosted on Workers AI were affected.

In total, 155 responses were found to be affected, all of which were associated with prompts exceeding 681,280 tokens. Cloudflare stated that the observed requests were consistent with ordinary long-context usage and found no evidence of deliberate exploitation. The company is publishing this report in coordination with SGLang, the open-source inference engine used for the model, to detail the bug, their response, and subsequent security hardening measures.

The root cause of the vulnerability stemmed from two overlapping issues within the SGLang inference engine. Firstly, a misconfiguration occurred where Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, but the Key-Value (KV) cache pool on the prefill GPUs was only configured to accommodate 681,280 tokens. While Cloudflare was aware of this mismatch, they expected the engine to return an error for oversized prompts, a behavior not observed in their long-context tests.

Secondly, a bug in SGLang's handling of these over-length requests in its disaggregated prefill/decode mode exacerbated the problem. When a prompt exceeded the KV cache pool's capacity, the engine internally marked it as rejected and replaced it with a placeholder. However, instead of aborting the request, the prefill process generated a KV cache for the placeholder and transferred it to the decode node. The decode node, receiving a 'transfer complete' message, incorrectly assumed the KV cache had been fully updated for the new request, when in reality, it retained stale data from previous customer sessions.

This resulted in the decode node generating responses that inadvertently included data from earlier customer requests, as it was attending over these stale cache pages. This cross-request KV cache exposure specifically occurred for prompts longer than the configured KV cache pool size, leading to the leakage of sensitive information.

Upon being alerted, Cloudflare acted swiftly. They disabled the affected model and implemented immediate remediation steps. To address the configuration issue, they adjusted the GLM 5.3 configuration to match their GLM 5.2 setup, increasing the KV pool size to approximately 2.04 million tokens, thereby ensuring that prompts fitting the context window would also fit the pool. This change alone prevents the bug from being triggered by valid requests.

Furthermore, Cloudflare patched the SGLang engine to make the rejection path more robust. This involved ensuring that requests with pending rejections are retired at the admission queue with their original error before any processing or KV transfer occurs. These fixes were validated against the researcher's reproduction script and Cloudflare's internal tests before the model was restored. The company continues to work with SGLang maintainers to address similar potential issues.

The incident highlights the complexities of managing large language models, particularly in multi-tenant cloud environments. The disaggregated architecture used for GLM 5.3, involving separate prefill and decode GPU pools, necessitates meticulous management of shared resources like the KV cache. While performance optimizations often involve overwriting memory pages between requests, this requires robust tracking of page ownership and lifetime by the inference engine to prevent data leakage, a critical lesson reinforced by this event.