RFC 9309: Robots Exclusion Protocol

This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how…

Read RFC 9309 on the RFC Editor

Outgoing relationships (9 related RFCs)

Normative reference — RFC 2119: Key words for use in RFCs to Indicate Requirement Levels
Normative reference — RFC 8174: Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words
Normative reference — RFC 5234: Augmented BNF for Syntax Specifications: ABNF
Normative reference — RFC 3986: Uniform Resource Identifier (URI): Generic Syntax
Normative reference — RFC 3629: UTF-8, a transformation format of ISO 10646
Normative reference — RFC 2046: Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types
Normative reference — RFC 9110: HTTP Semantics
Normative reference — RFC 8288: Web Linking
Normative reference — RFC 9111: HTTP Caching

Incoming relationships (1 related RFC)

Informatively referenced by — RFC 9969: Report from the IAB Workshop on AI-CONTROL