Skip to main content
Back to Blog

Ready-made robots.txt lines for AI crawlers, built from your settings

Rob, CEO & Founder3 min read

What is new

In AI Magnet, on the Advanced tools tab, a card now gives you ready-made lines for your robots.txt. HumanKey builds them from two settings you already have:

  • Bot Management: which AI crawlers you allow and which you block. For a crawler you have not set, HumanKey uses its default.
  • AI Content Rights: what AI may do with what it reads (search, AI answers, AI training). If you have not saved a choice, the card says which default it used.

The card is on every plan, Free included.

Lines to add, not a new file

The card gives you lines to add, not a whole file. You add them at the very end of your robots.txt. The rules already in the file stay and still apply.

Once HumanKey has read your file, the lines never:

  • add a blocking line for a crawler you allow;
  • block the crawlers that build search-engine indexes, such as Googlebot and Bingbot;
  • add a second group for a crawler your file already names. The card tells you to change your own group instead;
  • add a second set of rules for all crawlers (User-agent: *). If your file already has one without a Content-Signal line, the card shows the one line to put inside it by hand.

If your file already matches your settings, the card says there is nothing to add. If your site has no robots.txt yet, the lines are a complete starting file. If you saved a licence link on the AI Content Rights card, the lines end with a License: line. It is left out when your file already has it (more about that line).

Copy, download or send to IT

  • Copy puts the lines on your clipboard.
  • Download saves them as a text file.
  • Send to IT e-mails them to the person who runs your site, after you confirm you have the right to share that e-mail address.

The e-mail says plainly: do not replace the existing robots.txt. Add the lines at the very end, as they are, and keep every line already there. It also says that robots.txt is a request each crawler decides whether to follow, not a technical block.

Before you confirm your site

HumanKey reads your live robots.txt only after you confirm that the site is yours. Until then, the lines come from your settings alone, and the card explains where to put the Content-Signal line by hand. The card also tells you to change your own group if your file already has one for a listed crawler. You can copy or download these lines; Send to IT appears once HumanKey has read your file.

What the lines do

robots.txt is a request that crawlers which follow the rules respect, and Content-Signal declares rather than blocks. Read why we name every AI crawler and why we say "declare".

Where to find it

AI Magnet → Advanced tools → robots.txt: lines to add and comparison. On Pro and higher, the AI file package (Check and fix tab) builds robots.txt lines, an llms.txt file and tags for your homepage's head. It comes with install steps.

Questions about the card? Write to us through the contact form.

Know Your AI Traffic

Start tracking AI crawlers visiting your website today. Free for up to 1,000 checks per month.

Start Free Trial