Cloudflare Slashes AI Inference Costs by 68% Through Aggressive Optimization
Cloudflare details how it drastically reduced AI code reviewer inference costs by 68% through a multi-pronged optimization strategy, cutting expenses from $2.46 to $0.77 per review.
Cloudflare has announced a significant achievement in optimizing its AI code reviewer, successfully reducing inference costs by a remarkable 68%. The company detailed how it brought the cost per review down from an average of $2.46 to just $0.77, a feat accomplished through a focused effort on code optimizations, strategic model experimentation, and enhanced cache hit rates.
This aggressive cost-cutting initiative not only slashed expenses but also dramatically improved review efficiency. The median review time was reduced from over four minutes to a mere 1 minute and 32 seconds. This optimization was driven by two primary areas: approximately 40% of the savings stemmed from direct code and system optimizations, while the remaining 60% was achieved by experimenting with different AI models and fine-tuning their operational effort.
The cost per review had been steadily increasing since March, climbing from $1.22 to $2.32 by August. A contributing factor was the shift towards more expensive AI models, notably Anthropic's Claude Opus, which was adopted for the internal standards reviewer. This move, while improving performance on specific workloads, increased the cost per job from $1.38 to $2.03, with the internal standards reviewer alone jumping from $0.25 to $0.88 per review.
Furthermore, the AI reviewer's cache hit rate saw a substantial improvement, rising from 85.7% in April to an impressive 97% by September. Despite this, the number of tokens processed per review more than tripled during the same period, from 0.91 million to 2.99 million. This increase in token volume, even with a high cache hit rate, contributed to the escalating costs before the optimization efforts.
A significant portion of the cost was attributed to cache writes and reads, which accounted for 43% and 34% of the spend, respectively, with output tokens making up the remaining 23%. This indicated that a substantial amount of money was being spent on context that the model had already processed or was about to re-process. The AI reviewer was making an average of 16.6 large language model (LLM) calls per review, writing 72k tokens to the cache and reading back 1.08 million tokens.
To address this, Cloudflare focused on improving the effectiveness of Anthropic's prompt cache mechanism. This system caches parts of the prompt that have already been computed, allowing future requests to reuse earlier prefixes. The company identified that the harness tools, system prompt, and initial messages in their AI reviewer sessions rarely changed, making them ideal candidates for prefix caching.
The optimization strategy involved ensuring the first request of each session utilized the cache instead of writing new data, and reducing the overall number of calls per session. Data revealed that the first request of a session only found a cached prefix 14% to 26% of the time, despite the shared system prompt and tool definitions being identical across repositories. This was traced back to an issue in how OpenCode v1 constructed the system prompt, preventing the reuse of shared prompt components.
By implementing these changes, including optimizing the prompt construction and leveraging the prompt cache more effectively, Cloudflare managed to significantly reduce both costs and processing times. The company also noted the transition to OpenCode v2, a rewrite of their internal AI reviewer, which offered a 13% speed improvement for full-tier reviews, further contributing to the overall efficiency gains.