You Can't Block an AI Crawler From Reading Your Page. You Can Declare What It May Do With It.

Two different questions, one file

Every site owner who has looked at robots.txt knows the shape of it: User-agent, then Allow or Disallow. It answers exactly one question — may this crawler fetch this page at all? — and it has answered that question, for search engines, for three decades.

AI crawlers made a second question visible that robots.txt was never built to answer: once a crawler has fetched a page, what is it allowed to do with what it read? Feed it into search results? Feed it into an AI system's answer at the moment someone asks a question? Use it to train a model?

Those are three genuinely different uses, and a single Allow: / answers none of them. A site that allows a crawler to fetch a page has said nothing at all about which of those three uses it consents to — because robots.txt, as originally designed, has no vocabulary for the question.

A second, additive line — not a replacement

The fix is not a new gate that blocks anything. It's a second directive, declared alongside the existing Allow/Disallow rule, that says: of the crawlers I'm letting in, here is what I permit them to do with what they read. Three yes/no questions, answered explicitly:

• Search — may this be indexed and shown as a search result with a short excerpt?
• AI input — may this be used as input to an AI system answering a question or generating a summary, at the moment someone asks?
• AI training — may this be used to train or fine-tune a model?

A site can, for example, stay fully visible in search and citable by AI answer engines while declaring that training is not included in that grant — a position that was previously impossible to express in robots.txt at all. The three questions are independent; a site can answer any combination of them.

Why "declare" is the accurate word

This is the part worth being precise about, because the wrong word here creates a wrong expectation.

robots.txt — with or without this addition — is advisory. Nothing about adding a rights declaration next to your Allow: / line changes that. A crawler that ignores Disallow today will, mechanically, ignore this too; there is no new technical gate, no new 403, no new enforcement layer. If you want a crawler to be unable to fetch a page, that is still what Disallow is for, and it still only works against crawlers that choose to read the file.

So what does the declaration actually change, if not access? It changes the legal default — and that's not a smaller thing, it's a different mechanism entirely.

The legal hook: an EU opt-out, expressed in a file a crawler can read

Article 4 of the EU's Digital Single Market Copyright Directive ((EU) 2019/790) creates a broad exception: text and data mining — the kind of large-scale reading that trains a model — is permitted by default, without needing separate permission from each rightsholder.

Article 4(3) is the condition on that default: the exception applies only if the rightsholder has not expressly reserved their rights "in an appropriate manner" — and the Directive names machine-readable means as exactly that appropriate manner for content published online. Reserve your rights that way, and the exception no longer covers a use you didn't consent to; a party that mines your content for training anyway is back to needing an actual license, the same as they'd need for any other copyrighted work.

That is what a rights declaration in robots.txt is doing, legally: it is the machine-readable reservation Article 4(3) describes, sitting in the one place a crawler operator has no excuse not to have read. It doesn't summon an enforcement bot. It removes a legal default that used to run in the crawler's favor. "Declare," not "block," is simply what the mechanism is.

Two honest limits, stated plainly:

• This is EU law. Article 4(3) is the reservation mechanism under EU copyright rules. It reaches conduct EU copyright law reaches; it is not a universal, everywhere-enforceable switch.
• Nothing about a legal reservation makes a crawler technically incapable of ignoring it. A responsible crawler operator reads and respects the declaration because doing otherwise carries legal exposure under Article 4(3) — the same way a responsible crawler operator today already reads and mostly respects Disallow, out of a mix of law, reputation, and index-quality incentive, not because robots.txt can physically stop a request.

What the same declaration looks like in llms.txt

Sites that publish an llms.txt — the plain-language file some sites offer specifically for AI systems, separate from robots.txt — can restate the same declaration there in prose, so it reads consistently whichever file a system happens to check first. It's worth being clear about what llms.txt is not: it is a discovery aid, not a visibility mechanism. Publishing one doesn't rank you higher in an AI answer or guarantee you're read at all; it just means that if something does read your site, the file it finds is legible and says the same thing your robots.txt does.

What this is not

• Not a takedown tool. It doesn't retroactively affect anything already used to train an existing model. It's a forward-looking declaration about future crawling.
• Not enforcement. Nothing about publishing the declaration causes a request to be rejected. If blocking specific crawlers outright is the goal, that's a separate Allow/Disallow decision, made independently.
• Not a monetization mechanism by itself. A rights declaration says what's permitted, not what it costs. Licensing terms — if a site wants to attach any — are a separate, optional layer a site can point to from the same file; the declaration doesn't invent pricing on its own.
• Not a guarantee any given crawler honors it. It changes the legal default under EU law. Whether a specific crawler operator complies is, as with robots.txt generally, a matter of that operator's own conduct.

In practice

On HumanKey, the same three-question declaration is one control on the dashboard, generated straight into the robots.txt and llms.txt we produce for a connected site — findable-and-citable-but-not-trainable is the starting position, and it's a one-line change to widen or narrow it. The point of writing this post wasn't to introduce that control, though; it's that the underlying mechanism — an EU-law reservation expressed in a file crawlers already read — is worth understanding on its own, however a site chooses to publish it.

Reference points

• Directive (EU) 2019/790, Article 4 — the text and data mining exception and its rightsholder reservation mechanism (paragraph 3, machine-readable means)
• RFC 9309 — the Robots Exclusion Protocol, which defines robots.txt itself as advisory guidance for automated clients

---

HumanKey is an EU-based, GDPR-native analytics platform for publishers and e-commerce sites. This post explains a legal mechanism relevant to AI crawling and is not legal advice — if a rights-reservation strategy matters to your business, talk to counsel who can advise on your specific situation.

If you want to see which AI crawlers are actually reading your site before deciding what to declare, our free AI crawler report shows it in minutes.