Cloudflare Enhances Bot Trust with Continuous Verification System
Cloudflare introduces a new system to continuously evaluate bot behavior, moving beyond initial verification to build ongoing trust based on adherence to website preferences and directives.
Cloudflare has launched a new system designed to continuously assess the behavior of bots, even after they have been initially verified. This initiative aims to foster greater trust in automated traffic by moving beyond a one-time identity check to an ongoing evaluation of how consistently bots adhere to website rules and preferences. The core principle is that while verifying a bot's identity is crucial for accountability, true trust is earned through sustained, responsible actions.
The system focuses on two key questions: "Are you who you say you are?" and "Can we trust you?" While Cloudflare's Verified Bot program already addresses the first question through mechanisms like Web Bot Auth, the new system tackles the second by monitoring bot conduct over time. This addresses scenarios where a verified bot might still misbehave due to misconfiguration, bugs, or even intentional disregard for site rules, potentially overwhelming origins or evading controls.
A critical aspect of this continuous verification is the distinction between risk and trust. Risk is defined as the immediate likelihood of a harmful action, which can fluctuate. Trust, conversely, is built through repeated, consistent positive behavior. Cloudflare cites the example of Perplexity's bot being de-listed in 2025 for attempting to access resources on sites that had opted out of scraping and subsequently trying to bypass controls, illustrating how trust can be eroded by specific actions.
The system measures how consistently a bot respects site preferences, such as those outlined in a robots.txt file. It looks for repeated patterns of behavior that disregard these directives, rather than isolated incidents. This includes checking if bots consult and honor the preferences published by website operators, aligning with standards like RFC 9309. The goal is to ensure that bots do not crawl or access resources in ways that website owners have explicitly forbidden.
Furthermore, the system evaluates how bots respond to direct feedback from websites, particularly HTTP 429 "Too Many Requests" responses. These responses serve as directives for bots to reduce their request rate. The new Cloudflare system monitors whether bots adapt their behavior accordingly – slowing down or pausing their requests – or if they continue at the same pace, ignoring the signal. Consistent adherence to these signals increases a bot's trust score, while repeated disregard lowers it.
These observations will be translated into an "explainable trust band," providing a range that indicates how consistently a bot respects site preferences and the specific behaviors that informed this assessment. This offers a more nuanced view than a permanent judgment on the bot or its operator. Cloudflare plans to share these scores publicly on Cloudflare Radar and expand the metrics used for evaluation.
This continuous verification system, combined with existing tools like Bot Preference Sync and Bot Management, forms a comprehensive accountability loop for automated traffic. Bot Preference Sync allows website owners to communicate their preferences, while the new system observes how bots interact with them. Bot Management then enables site owners to act on this information, ensuring that trust is earned through transparent and accountable behavior, not merely assumed from initial verification.
Cloudflare is currently using this system internally to refine its evaluation methods and build a history of bot behavior. The company plans to make this system available to all Cloudflare customers, allowing them to set their own trust thresholds for managing Verified bot traffic. The ultimate aim is to keep website preferences meaningful and ensure that automated traffic operates in a manner that respects site owner intentions.