Introducing Web Search API via AI Gateway
Cloudflare integrates Web Search API into AI Gateway, enabling agents to fetch live web data via partners Ceramic.ai, Exa, and Linkup while enforcing crawler transparency standards.
Agents traditionally guess URLs to fetch web content, often resulting in failed requests like 404 errors. Cloudflare’s new Web Search API addresses this by allowing agents to search the internet like humans do, retrieving live, relevant data instead of relying on static, outdated information. The integration partners with Ceramic.ai, Exa, and Linkup to provide grounded intelligence through AI Gateway, ensuring agents access fresh, structured snippets from the web. This dynamic context layer helps models stay current with recent events, evolving APIs, and fast-changing news, improving response accuracy.
Cloudflare emphasizes transparency and control for web crawlers, requiring partners to comply with its "Verified bots" standards. These standards mandate that crawlers respect robots.txt, identify themselves, and include source links in search results. The initiative aims to foster a fair internet by giving site owners meaningful control over their data usage. Partners have committed to these rules, reinforcing Cloudflare’s role in setting higher standards for ethical web crawling.
AI Gateway serves as the control plane for integrating Web Search API, offering unified billing, security, and access controls. Developers can consume web search using AI Gateway credits, monitor requests via observability logs, and manage access to different providers. Zero Data Retention (ZDR) partners are clearly identified, ensuring user data is not retained. Web search is available at list API pricing without additional markup, and Bring-Your-Own-Key (BYOK) is supported for seamless integration.
Integration is flexible across platforms: developers can query web search via a REST endpoint using an AI Gateway token or use a simple line of code for Cloudflare Workers. Native Server Tools in AI Gateway will soon include web search, reducing the need for manual tool orchestration. A standalone worker binding is also available for immediate use. The feature is accessible via AI Gateway, standalone REST API, or future formats, with documentation and an AI Playground for testing.