Robots.txt
Website file expressing crawler access preferences through the Robots Exclusion Protocol.
Standardised in RFC 9309 in 2022, Robots.txt lets website operators tell crawlers which paths they may fetch. The protocol communicates instructions; it does not enforce access restrictions.
Last refreshed: 22 October 2026 · Appears in 1 active topic
Timeline for Robots.txt
Mentioned in: Australian broadcasters press AI to pay
Media's AI PivotMentioned in: CMA orders a Google AI opt-out tool
Media's AI PivotBackground
Website operators use Robots.txt text files to communicate crawling rules. Under the Robots Exclusion Protocol, an operator places the file at the service’s top-level /Robots.txt path. Rules identify a software agent and specify which paths it may or may not fetch. A cooperating crawler reads those instructions when deciding where to go. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html).
The protocol helps website operators communicate crawling preferences to automated clients. The standard explicitly separates its rules from access authorisation and security controls. A Robots.txt entry does not physically prevent a client from requesting a page. This makes the file useful for expressing crawling preferences, including preferences directed at particular agents, while leaving enforcement to other measures and the behaviour of the crawler.
What does robots.txt tell a web crawler?
How does robots.txt stop unwanted copying?
Where does a website publish robots.txt?
Written with AI from Lowdown's published reporting. Not individually reviewed by an editor. Updated 22 October 2026. Editorial standards.