Cloudflare Promised Isolation. It Tested Only Status Codes.

– views

Source: Cloudflare — Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang (26 October 2026)

  • Cloudflare
  • Workers AI
  • GLM-5.3
  • multi-tenancy
  • KV cache
  • data governance

Cloudflare disclosed on 26 October 2026 that its Workers AI deployment of Z.ai’s GLM-5.3 (@cf/zai-org/glm-5.3) had leaked content across tenants: 155 responses, “all of which were to requests with prompts longer than 681,280 tokens”, returned output that contained content from other customers’ requests. The exposure ran from 27 August to 9 September 2026, from the moment the model launched.

The remediation was fast. The interesting part is how the bug survived launch. In Cloudflare’s own words, it knew before release that the configured context window was larger than the GPU memory serving it, and its long-context test checked the HTTP status code rather than the answer. Disclosure: this site runs on Cloudflare Workers and D1 but does not use Workers AI; we are not a party to the incident.

The two numbers that were never forced to agree

Cloudflare’s post gives both figures:

Tokens
GLM-5.3 configured context window1,048,576
Prefill KV cache pool, as deployed681,280
Same pool after remediation~2,040,000

A request could fit the advertised window and exceed the memory that actually serves it. Cloudflare’s explanation is more interesting than a misconfiguration, because the pool is not a knob:

“Furthermore, the size of this pool is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after loading the model weights. The model’s advertised context length is a separate configuration value. Nothing forces the two to agree, and it is our job to check that they do.”

The pool size is in the engine’s startup log and exposed as a metric. It was visible; it was never compared to the config.

The assumption the incident is made of

“In our pre-launch testing of this model, we observed successful HTTP status codes and a streaming body with no error frames. We assumed that the engine was capable of gracefully handling the difference between the KV cache pool and requested context window by returning an error. This was one critical gap in our testing: we only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence.”

The over-length request did not fail. It succeeded: “The client got HTTP 200 instead of HTTP 400”, and the response arrived faster than a 681,280-token prefill could plausibly take.

SGLang’s own disclosure, published the same day, makes the assumption legible. In the setup Cloudflare shipped, a request marked for rejection still completed the prefill-to-decode handoff on a one-token placeholder, and decode read cache pages never populated for it. In the other architecture it did fail: “The single-node rejection path returned an error instead.” The graceful behaviour existed — not in the deployment Cloudflare tested. Once the model went private on 9 September, the same over-length requests “return HTTP 400”.

The sentence the leak breaks

Workers AI’s data page, last updated 16 October 2026 — five weeks after the incident was fixed, and it does not mention the incident — still says:

“Your content remains yours: You own, and are responsible for, your Customer Content. Cloudflare does not make it available to other Cloudflare customers.”

and

“For standard inference requests that do not invoke a stateful feature, Cloudflare does not retain inputs or outputs after they are no longer needed to process the request and deliver the response.”

Both sentences are about what Cloudflare does with your content. The failure was in what the runtime handed to someone else while processing it. SGLang describes that direction precisely: “the realistic impact is therefore one-way cross-request leakage, in which fragments of the aborted request’s context can surface in another client’s output”, and “a deployer has no reliable way to tell” whether it already happened, because the corruption leaves no error or log line.

GPU memory is not zeroed, and that is the default

“Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it. This approach is common among inference engines and is necessary to achieve the best performance, but it means that the inference engine is responsible for tracking ownership of these memory pages.”

Isolation between tenants on one GPU is therefore a bookkeeping claim, not a memory-boundary guarantee. Cloudflare lists zeroing pages on release as a draft proposal, and prices it: “our benchmarks showed a 10-20% lower overall throughput”. Its two detection tools, --disaggregation-enable-kv-checksum and --kv-canary, “do not yet cover every model architecture”, and it is working with SGLang “to extend both checks to more architectures, including GLM 5.3” — the model this incident was about.

And the fix is not the end: “Relying on pool size alone is fragile”, since the pool “can shrink after deployment as weights, quantization settings, parallelism, or other memory allocations change”. GLM-5.3 was the only Workers AI model affected, because the other disaggregated deployments happened to have pools larger than their context windows.

What a reader can actually do with this

If you call a hosted model API, the published context length is a claimed ceiling, not a measured one. Three cheap checks: send a prompt past the advertised limit on a staging key and read the body, not the status; ask whether prefill and decode run on separate machines, the shape this bug class needs; ask whether output correctness, not just latency and status, is in their long-context test suite. Self-hosting SGLang in that split configuration? Its guidance is v0.5.22 or later, with checksum and canary enabled where your architecture supports them.

This is the second time this site has reported a Cloudflare statement about data that the runtime did not enforce: the “Disallow AI Training” directive is a declaration, not a mechanism. Different layer, same lesson — a policy sentence and a memory-safety boundary are maintained by different teams, and only one of them is in the request path.

What we could not verify

  • Whether the affected customers were told. Cloudflare publishes a timeline from report to fix, and says 155 responses were affected, but not whether those accounts were notified individually, nor whether any regulator notification was made.
  • What the leaked content was. “No evidence of deliberate exploitation” is stated; sensitivity, customer identity and token volume beyond those 155 responses are not.
  • The researcher’s writeup. SGLang’s post says it is “available soon”. As of today it is not published, so every number here comes from Cloudflare and SGLang, not from the finder.
  • Today’s real ceiling for GLM-5.3. The model page still advertises 1,048,576 tokens with no second limit, at $1.40 / $4.40 / $0.26 per million. The ~2.04M pool figure comes from the incident post, not from a spec.

The bottom line: the isolation promise in Cloudflare’s data page is enforced by a bookkeeping layer that Cloudflare’s own pre-launch test never exercised. A status code of 200 was the only evidence it had.

More coverage