Say it once: introducing Bot Preference Sync
Cloudflare has launched Bot Preference Sync, a tool that automatically aligns robots.txt preferences with AI bot policies set in its dashboard, aiming to resolve inconsistencies between stated rules and enforced protections.
Cloudflare’s new Bot Preference Sync feature addresses a longstanding issue where conflicting rules in robots.txt and enforcement layers could undermine site owners’ preferences regarding AI crawlers. The tool automatically updates robots.txt to reflect AI bot policies—such as allowing or blocking Search, Agent, or Training traffic—set in the Cloudflare dashboard, ensuring consistency between stated and enforced rules. Previously, discrepancies between these layers could lead to crawlers disregarding site owners’ preferences or attempting to bypass restrictions. The feature is now available to all customers, from Free tier to Enterprise, and can be toggled on or off at any time.
The announcement follows Cloudflare’s July 1, 2026, introduction of options to manage AI traffic use cases, including Search, Agent, and Training categories. Bot Preference Sync simplifies the process by generating or updating robots.txt based on these configurations, prepending new directives to existing files while preserving prior Disallow rules. For publishers monetizing through ads, the feature defaults to disallowing Training traffic unless manually adjusted, aligning with their business needs. New customers without legacy configurations will start with no default blocks, allowing them to set policies from scratch.
To enhance transparency, Cloudflare requires mixed-use crawlers—those blending Search, Agent, and Training functions—to provide additional verification when site owners disallow Training. Bots meeting these criteria are listed in Cloudflare Radar’s AI bot transparency section, while those failing to comply risk being blocked despite other allowed categories. This approach aims to reward operators who clearly disclose their data usage practices, reinforcing accountability in the AI crawler ecosystem.
Bot Preference Sync is part of Cloudflare’s broader effort to give site owners greater control over how their content is accessed and used by AI systems. The feature will roll out to all customers in the coming week, with prompts in dashboards and inboxes to confirm preferences. Cloudflare emphasizes ongoing collaboration with major bot operators to address challenges like unauthorized training and evolving questions around discoverability and engagement, while maintaining a focus on transparency and user control.