Making it easier for websites to express AI training preferences

OpenAI announced it will recognize a common signal embedded in robots.txt files, which websites use to set preferences for automated access. The signal allows publishers to communicate a single instruction to multiple AI training providers simultaneously, rather than managing separate rules for each. This voluntary approach aims to simplify compliance for website operators who wish to restrict generative AI training on their content. The company plans to implement the signal in the coming months and will provide further details at that time.

The new signal will operate alongside OpenAI’s existing crawler controls within robots.txt, which already allow publishers to specify access rules for their pages. These existing controls will remain unchanged and continue to be respected by OpenAI. The shared signal is part of ongoing efforts by the Internet Engineering Task Force’s AI Preferences working group to standardize web-based preferences for AI training. OpenAI is joining other participants in this initiative.

The move reflects OpenAI’s participation in IETF discussions to establish clearer, consistent standards for how website preferences are communicated to AI systems. The company emphasized that its practices will evolve in response to ongoing conversations and changes in the broader landscape. OpenAI also called on other AI providers to adopt the shared signal to improve clarity and consistency for website operators.

OpenAI framed the announcement as a step toward maintaining an open web where publishers can control how their content is used while ensuring discoverability. The company highlighted the importance of providing website owners with simpler, more unified ways to express their preferences about AI training. By supporting the shared signal, OpenAI aims to reduce fragmentation in how preferences are handled across the industry.