Your robots.txt recommendation should name every AI crawler — not just the ones you have already seen

Why this post exists

This is a product-transparency post about a principle we hold ourselves to, not an announcement of a brand-new feature. If you run a website, an AI crawler you have never heard of can start visiting you next month. So a sharp question to ask any "AI crawler control" tool is: when a new crawler shows up, will you tell me it exists and let me decide — or will I only find out after it has already been reading my site for weeks?

Our answer is a completeness rule. A recommendation about which crawlers to allow or block should name every AI crawler we track, not only the handful that already appear in your traffic. This post explains why, in plain language, and why we treat completeness as a matter of honesty rather than a nice-to-have.

The incompleteness trap: recommending only what you have already seen

The easy way to build a robots.txt recommendation is to look at your recent traffic, list the crawlers you have actually been visited by, and suggest a rule for each one. It looks helpful, and for the bots in your logs it is.

But it has a blind spot that is the exact opposite of what you need. The crawler you most want to make a deliberate decision about is often one you have not seen yet — a newly launched AI assistant, a search crawler for an engine you do not use, or an advertising crawler that will start arriving once a campaign goes live. A recommendation built only from your logs cannot mention any of those, because by definition they have not shown up. It can only ever tell you about the past.

Leaving them out is not neutral. It quietly narrows your choices to whatever already happened, at exactly the moment you would benefit most from getting ahead of what is coming.

Completeness as honesty, not just coverage

We treat completeness as a fairness property, not a feature checkbox.

Under the EU framework for fair commercial practices, leaving out information a buyer materially needs can itself be misleading — even when everything you did say is true. A tool that promises "complete control over AI crawlers" and then silently omits crawlers is, in a small way, contradicting its own promise. The honest posture is the harder one: show you the whole set we track and let you decide, rather than showing you a tidy shortlist and hoping you do not notice what is missing.

So our recommended robots.txt names every AI crawler we track — 60+ of them — each with a suggested posture, whether or not it is in your recent logs. If you would rather ignore most of them, that is your call to make with the full picture in front of you.

What this looks like when the advice is written by a model

We also offer an AI advisor that reads your traffic and writes recommendations in plain language. It now follows the same completeness rule.

The important part is how it handles a crawler you have not been visited by. It does not pretend. It will suggest an explicit rule for a high-value crawler that has not shown up yet, and it says so honestly — "not seen in your traffic yet, but worth an explicit rule" — rather than inventing a visit count to justify the suggestion. Real numbers describe real traffic; a proactive suggestion is labelled as exactly that. Completeness and honesty have to travel together, or the first one becomes a licence to make things up.

Telling two things apart: your settings versus your live file

There is a subtler honesty point hiding in this space, and it is easy to get wrong.

"Blocked" can mean two different things. A crawler can be blocked in your settings inside our product, or blocked in the actual robots.txt file your website serves to the world. Those are not the same thing, and they can disagree — you might have a crawler marked one way in a dashboard while your live file says the opposite.

Good advice has to keep them distinct. Our recommendations are careful to tell you which one they mean, so you are never told a crawler is "blocked in your robots.txt" when the file your site actually serves is letting it through. Conflating the two would be an easy way to sound authoritative while being wrong.

Advisory, never automatic

None of this touches your site by itself. Every recommendation is advisory: guidance for you to review, not an action taken on your behalf. We mark AI-generated guidance clearly as machine-generated, in line with the EU AI Act's transparency requirement, and nothing is applied until you choose to apply it.

It is worth remembering that robots.txt is itself advisory. Well-behaved crawlers honour it; others ignore it. That is precisely why seeing the complete picture matters: the file is a statement of intent, and a statement of intent is only useful if it reflects a decision you actually made about every crawler that matters — not just the ones that happened to arrive first.

We hold our own site to the same rule

We use this on ourselves. Our own robots.txt is generated from the same list of AI crawlers we track, so it stays complete on its own as new crawlers are added — we welcome the assistants and search crawlers, and we turn away the small set of aggressive scrapers, exactly as our product would recommend. If we are going to tell you that a crawler-control file should be complete, the least we can do is keep ours that way too.