Cloudflare Patches Cross-Tenant Data Exposure in Containers Service
Cloudflare has fixed a vulnerability in its Containers and Sandboxes services that could have allowed a customer to recover residual data from previous containers on the same host.
Cloudflare has successfully remediated a cross-tenant data exposure vulnerability affecting its Containers and Sandboxes services. The issue, reported responsibly by security researcher Oren Yomtov of Accomplish, stemmed from a configuration that disabled block zeroing on shared storage pools.
Cloudflare Containers operate on multi-tenant infrastructure, meaning customers do not control the underlying physical servers. The vulnerability allowed a customer with a Workers Paid account to potentially recover residual disk blocks from containers that had previously occupied the same host. However, the exploit was not targeted, meaning it could not aim for a specific customer, workload, or data, and the presence of residual data was not guaranteed.
The core of the problem lay in how Cloudflare Containers managed storage. They utilize Linux device mapper thin provisioning, where physical storage is allocated only when a virtual disk writes to a new region. The affected storage pools were configured with skip_block_zeroing enabled. This setting meant that when a physical block was reassigned from one container to another, only the portion written by the new container was zeroed. The remaining parts of the block could retain data from its previous occupant.
The proof-of-concept demonstrated by the researchers involved creating a container, writing a small amount of data (4 KiB) into specific regions of its root disk, and then performing a raw-device read. Because block zeroing was disabled, the unwritten portions of the reallocated 64 KiB block could still contain data from a prior container. The researchers used ext4 filesystem checksums to differentiate their own test data from any residual foreign data they might recover.
To validate the vulnerability, the researchers tested their method across multiple production placements. They reported observing residual material on 18 out of 24 placements and 20 out of 22 underlying nodes across different continents. The recovered data included filesystem metadata, directory structures, and even complete SQLite databases. Importantly, the researchers confirmed that the recovered data remained confidential and was securely deleted after submission, adhering to Cloudflare's bug bounty policies.
Cloudflare has deployed a fix across its entire Containers fleet, requiring no action from customers. The company has thoroughly reviewed its historical disk I/O telemetry and found no evidence of malicious exploitation. All observed activity attributable to the vulnerability was traced back to the researchers' validation efforts and Cloudflare engineers testing the fix.
While the vulnerability could have potentially allowed for data leakage across tenant boundaries, Cloudflare emphasized that the exploit was not targeted and recovery was not guaranteed. The company's swift response and comprehensive remediation demonstrate a commitment to maintaining the security and isolation of its multi-tenant services.
This incident highlights the ongoing challenges in managing data persistence and isolation in cloud-native environments, particularly when leveraging advanced storage features like thin provisioning without explicit zeroing mechanisms.