Cloudflare Introduces Meerkat: An Experimental Global Consensus Service Powered by QuePaxa
Cloudflare's Research team has developed Meerkat, an experimental global consensus service utilizing the QuePaxa algorithm to manage control-plane state across its data centers.
Cloudflare is rolling out Meerkat, an experimental service designed to manage critical control-plane state across its extensive network of over 330 global data centers. This new system leverages the QuePaxa consensus algorithm, a departure from more traditional algorithms like Raft, which have historically presented challenges in wide-area network environments.
Many of Cloudflare's internal services require consistent access to and modification of control-plane data, which dictates how these services operate. This data includes information like resource placement and leadership assignments for replicated databases. Ensuring strong consistency—meaning all readers see the same data in the order it was written—is paramount, especially given the unpredictable nature of the internet where network links can fail and data centers can become temporarily unavailable.
Traditional consensus algorithms such as Raft often rely on a single leader to manage writes. If this leader fails or becomes unreachable due to network issues, the entire system can become unavailable until a new leader is elected. This process, often dependent on timeouts, can be difficult to tune in networks with variable latency, leading to incidents of unavailability that Cloudflare has experienced.
QuePaxa, the algorithm powering Meerkat, offers a distinct advantage by allowing all replicas to perform writes concurrently. Crucially, it avoids halting progress due to timeouts, making it a more robust choice for Cloudflare's global infrastructure. This design aims to maintain availability even when faced with network disruptions or server outages.
Meerkat is built to provide linearizability, a strong consistency model that simplifies programming by allowing developers to reason about the distributed system as if it were a single machine. This eliminates the complexities associated with weaker consistency models where writes can be reordered, potentially leading to unexpected application behavior.
While Meerkat is still under development and currently intended for internal use, its deployment represents a significant milestone. To the best of Cloudflare's knowledge, this marks the first industrial-scale implementation of the QuePaxa algorithm. The service is initially focused on managing smaller pieces of control-plane state.
Future posts from Cloudflare are expected to delve deeper into the technical specifics of Meerkat, including its implementation of serializability and other advanced features. The development of Meerkat underscores Cloudflare's ongoing commitment to building resilient and highly available distributed systems capable of operating reliably across the global internet.