Cloudflare and OpenAI: making AI discovery better for everyone

Optimizing a website for AI systems is only effective if those systems can locate the content in the first place. Many pages—such as installation guides or product reviews—contain valuable information that may never be found by traditional search engines. As users increasingly rely on AI tools like ChatGPT for answers, ensuring these systems can discover and accurately represent such content becomes critical. The pilot, launched in July 2026, aimed to test whether Cloudflare’s network signals could enhance OpenAI’s ability to index relevant pages and direct users to them.

The pilot, involving roughly 1% of Cloudflare’s network, allowed OpenAI’s crawler to identify and index hundreds of millions of previously undiscovered web pages. These pages were then evaluated for relevance and utility before being surfaced in ChatGPT responses. The collaboration also introduced mechanisms to filter out low-quality or inaccessible pages, reducing unnecessary crawling and improving efficiency for both website owners and AI systems.

Early findings revealed patterns that could mislead crawlers, such as bots requesting non-existent files or sites redirecting all URLs to a homepage. Cloudflare and OpenAI developed signals to detect these issues, preventing wasted resources and improving the accuracy of AI-driven discovery. Additional signals were tested to flag harmful or misleading pages, ensuring safer results without penalizing entire domains for isolated issues.

The next phase of the project will focus on refining page selection, enhancing change detection signals, and measuring the real-world impact on website traffic and user engagement. Domain owners retain control over how AI systems access their content, and Cloudflare’s AI Crawl Control tool allows them to manage these interactions. The research prioritizes discovery over content extraction, using metadata rather than page content for model training.