RSL: A Second Line in robots.txt, Pointing at a Document You Host

A directive that points, rather than declares

robots.txt already answers two questions a site can ask a crawler: may you fetch this page at all (Allow/Disallow), and — for crawlers that read it — what may you do with what you read (a compact, three-value rights declaration for search, AI input, and AI training, sitting in the same file).

A newer, separate line — part of a standard called RSL (Really Simple Licensing) — answers a third, different kind of question: is there a document, somewhere, that spells out actual licensing terms for this content? Instead of encoding an answer directly in robots.txt, it points — a License: line naming a URL. What lives at that URL is entirely up to the site: a plain terms page, a machine-readable license file, a pricing page, whatever the site wants a crawler (or a human reading the source) to find.

That's the whole mechanism. It's a pointer, not a payload.

Why a pointer, and not just more directives

The rights-declaration line already covers three fixed yes/no questions — useful precisely because it's fixed and short enough to sit in every crawler group of a file that's meant to be read in milliseconds. But "yes/no on training" and "here are my actual licensing terms, including price" are different amounts of information. Terms, conditions, and pricing don't fit in a one-line directive, and trying to force them into one would mean inventing new syntax every time a site wanted to say something the fixed vocabulary didn't cover.

A pointer sidesteps that. The site publishes whatever document it wants, in whatever format it wants, and updates it on its own schedule — without anyone needing to agree on a new file format first.

The honest state of adoption

This standard is genuinely new: version 1.0 was finalized in December 2025. On the publisher side, adoption moved fast — well over a thousand sites and media organizations added support within months of launch.

What's not yet established is the other side of that exchange: whether AI crawler operators actually fetch the document a License: line points to, read it, and change their behavior based on what it says. As of today, no major AI crawler operator has publicly confirmed that its crawler does this. That's a meaningfully different state than robots.txt's Allow/Disallow, which the largest crawlers have said for years that they respect.

So the honest way to describe adding a License: line today: it costs nothing, it's easy to add, and it puts your terms in a place a crawler — or a human, or a future crawler that does start checking — can find. It is not yet a signal with confirmed uptake on the other end. Treat it as a declaration you're making available, not a lever anyone is confirmed to be pulling.

How this differs from the rights declaration next to it

It's worth being precise about the difference, since both lines can sit in the same file:

• The rights declaration (search / AI input / AI training) is a fixed, compact, yes/no signal, tied to a specific EU legal mechanism for reserving rights in a machine-readable way. It says nothing about price or conditions — only permission.
• The license pointer is open-ended. It can lead to anything from a one-sentence "contact us for a license" page to a fully worked-out pricing document. It carries no legal weight of its own; whatever weight it has comes from the document it points to and whatever agreement, if any, a crawler operator enters into after reading it.

They're complementary, not competing: a site can decline AI training in the compact declaration while the pointer leads to a page saying "training rights are available for a fee, contact us" — the declaration sets the default, the pointer offers a path around it.

What this is not

• Not a paywall. Nothing about publishing the pointer blocks a request. A crawler that never checks it behaves exactly as if it weren't there.
• Not an authoring tool. The document behind the link is something the site itself writes and hosts — not a license HumanKey drafts, negotiates, or is a party to.
• Not a substitute for the rights declaration. If what a site wants to say is a simple yes/no on training, search, and AI input, the compact declaration already says that, in a form every crawler group can carry.
• Not a guarantee anyone reads it. See the adoption section above — this is the part worth repeating rather than glossing over.

In practice

On HumanKey, adding a license pointer is one optional field on the same dashboard control as the rights declaration — a URL to a document the site already hosts. It's additive: if it's not set, robots.txt looks exactly as it did before. If it is set, it appears as one line, unchanged in wording by anything else on the page.

Reference points

• RSL 1.0 Specification (rslstandard.org) — the technical definition of the license-pointer directive discussed here

---

HumanKey is an EU-based, GDPR-native analytics platform for publishers and e-commerce sites. This post explains a technical publishing mechanism and current market adoption; it's not legal advice on licensing your content — talk to counsel if a licensing strategy matters to your business.

If you want to see which AI crawlers are actually reading your site today, our free AI crawler report shows it in minutes.