OpenSSL 'HollowByte' Vulnerability Allows DoS with 11-Byte Payload
A newly discovered OpenSSL vulnerability, dubbed 'HollowByte,' enables unauthenticated remote attackers to cause a denial-of-service by sending a minimal 11-byte payload, exploiting memory allocation flaws during TLS handshakes.
A critical vulnerability in the widely-used OpenSSL cryptographic library, nicknamed 'HollowByte,' has been disclosed, allowing unauthenticated remote attackers to trigger a denial-of-service (DoS) condition with an exceptionally small malicious payload of just 11 bytes. The discovery, attributed to the Okta Red Team, exploits a fundamental flaw in how OpenSSL handles memory allocation during the initial stages of a TLS handshake, before any authentication or data validation occurs.
The vulnerability hinges on the way older OpenSSL versions process the ClientHello message. This message includes a 4-byte header that declares the size of the incoming message body. The library, trusting this declared size, pre-allocates a receive buffer. When an attacker crafts an 11-byte payload, this header tricks OpenSSL into allocating a buffer of up to 131 KB based solely on the attacker's declared size. The subsequent processing then blocks indefinitely, waiting for data that will never arrive, effectively consuming server resources.
Beyond simply holding connections open, HollowByte introduces a more insidious problem: memory fragmentation. When an attacker's connection is terminated, the allocated buffer is freed. However, the underlying memory management system, such as glibc, does not immediately return these smaller-to-medium sized allocations to the operating system. Instead, it retains them for potential reuse. By launching numerous connections with varying claimed buffer sizes, attackers can prevent the allocator from efficiently reusing these freed memory chunks. This leads to a continuous and permanent climb in the server's Resident Set Size (RSS), consuming system memory even after the attack ceases. The only effective remedy for this state is to terminate the affected process.
Testing by Okta Red Team demonstrated the severe impact of HollowByte. In a 1 GB RAM environment, an unpatched OpenSSL instance running NGINX was rendered unusable after accumulating 547 MB of frozen, fragmented memory, eventually being killed by the Out-Of-Memory (OOM) killer. Even in a larger 16 GB RAM environment, the attack managed to lock up 25% of total system memory while staying below typical connection limits, rendering standard connection-limiting defenses ineffective.
The blast radius of HollowByte is extensive due to OpenSSL's ubiquitous presence in internet infrastructure. The vulnerability affects a wide array of critical components, including popular web servers like Apache and NGINX, language runtimes such as Node.js, Python, Ruby, and PHP, and databases including MySQL and PostgreSQL. Any system relying on vulnerable versions of OpenSSL for TLS termination is potentially at risk.
OpenSSL has addressed the HollowByte vulnerability through a series of pull requests that modify the buffer allocation logic. Instead of blindly trusting the declared size in the header, the library now employs incremental buffer growth, expanding the buffer only as data actually arrives. This ensures that an attacker's initial claim costs the server minimal resources. The fix has been integrated into OpenSSL version 4.0.1 and has been backported to maintenance branches, including 3.6.3, 3.5.7, 3.4.6, and 3.0.21.
Notably, OpenSSL has treated this issue as a hardening improvement rather than a critical vulnerability warranting a formal CVE advisory. This approach, where DoS-class issues are patched silently, means that many organizations may overlook HollowByte if they rely solely on CVE scanning for vulnerability management. Given the critical nature and widespread impact, security teams are strongly advised to prioritize updating their OpenSSL installations to the patched versions, regardless of the absence of a CVE designation.
Defenders should immediately upgrade to OpenSSL 4.0.1 or the relevant backported version. Furthermore, it is crucial to audit and update bundled OpenSSL versions within language runtimes and applications, as OS-level patches may not cover these instances. Monitoring the RSS memory usage on TLS-terminating servers for unexplained gradual increases can also serve as an early warning sign of potential fragmentation attacks.
The OpenSSL team has silently addressed the HollowByte vulnerability, assigning no specific CVE identifier but backporting the fix to several older, still-supported versions including 3.6.3, 3.5.7, 3.4.6, and 3.0.21. While Okta's researchers recommend immediate upgrades, OpenSSL itself has classified the fix as a 'hardening' measure rather than a security vulnerability, implying it was addressed through improved buffer allocation logic that now waits for data before allocating memory, rather than trusting header claims.
This new report provides further technical details on the HollowByte flaw, explaining how the vulnerability exploits glibc's memory allocation behavior to cause heap fragmentation that persists even after the connection is dropped. It also highlights that the fix was applied without a CVE, advisory, or changelog entry, making it difficult for organizations to track and patch, and notes that the DTLS path remains unpatched.