Cloudflare Hyperdrive MySQL Caching Bug Linked to Character Encoding Issues
Cloudflare has detailed a rare MySQL caching bug in its Hyperdrive database connection pooler, stemming from character set encoding negotiation problems.
Cloudflare's Hyperdrive, a database connection pooler designed for its Workers platform, has achieved general availability for MySQL protocol support after a beta period that involved extensive bug fixing. Initially supporting only PostgreSQL, Hyperdrive's expansion to MySQL required addressing numerous issues to meet a 99.99% production availability target. During this process, a subtle but impactful bug related to character set encoding was discovered, affecting high-volume customers.
Hyperdrive functions by maintaining open SQL connections globally and sharing them across multiple Worker requests, thereby reducing latency and database load. Establishing a new database connection is resource-intensive, involving multiple network round trips. For serverless platforms like Workers, connection pooling is essential to avoid impractical latency and high per-connection costs. While MySQL is more efficient than PostgreSQL in its process management per connection, large-scale deployments can still face resource constraints.
One of the initial bugs reported during the beta involved emojis being incorrectly returned from the database. This was traced to an issue with character set encoding negotiation during the connection handshake. Hyperdrive initially negotiated 'utf8', which is a deprecated alias for 'utf8mb3'. This encoding only supports up to three bytes per Unicode code point, falling short of the full UTF-8 standard which requires up to four bytes per code point. The resolution involved updating the handshake to use 'utf8mb4', which correctly handles all UTF-8 characters, including emojis.
A subsequent encoding-related problem emerged after the implementation of connection cleanup using the COM_RESET_CONNECTION command. This command is intended to clear session state on a connection before it is reused. However, it was found that COM_RESET_CONNECTION does not revert the connection to the initially negotiated character set. Instead, it resets to the server's default encoding. For instance, if a MySQL server defaults to 'latin1' but the connection was negotiated as 'utf8mb4', running COM_RESET_CONNECTION would revert the encoding to 'latin1', breaking subsequent operations. The fix involved explicitly re-issuing a command to set the encoding to 'utf8mb4' after each connection reset.
The most challenging bug, however, manifested as a small but consistent rate of empty query responses where a non-empty response was expected. This issue was particularly difficult to diagnose due to its low occurrence rate (approximately 0.001%) and was only reported by the highest-volume MySQL customer. Extensive telemetry analysis and increased tracing provided few clues, and network issues were ruled out.
A critical clue was that the bug only occurred on Hyperdrive instances with caching enabled. This pointed towards an interaction issue between Hyperdrive's caching mechanism and the MySQL protocol. Hyperdrive employs a split architecture with Client and Endpoint nodes, both utilizing local CDN caches. When nodes are in the same data center, they share a cache, which is generally beneficial for performance. The caching mechanism relies on hashing normalized wire message contents to store response bytes.
Further investigation into the MySQL client/server protocol revealed that assumptions made based on the PostgreSQL protocol might have contributed to the problem. The exact nature of the interaction between the shared caching, the COM_RESET_CONNECTION command, and the character set handling within the MySQL protocol was the focus of the deep dive to pinpoint the root cause of the unexpected empty query responses.