Skip to main content
Back to Blog

Free homepage check: see what AI crawlers read on your site

Rob, CEO & Founder5 min read

The page a crawler sees is not always the page you see

When you open your homepage in a browser, the browser runs the page's scripts and builds what you see. A crawler that does not run scripts sees only the HTML your server sends. The two can be quite different: a title, a description or a whole block of text may exist only after the scripts have run.

That gap is easy to miss, because you rarely see the raw HTML yourself. So we built a check that shows it to you, for public websites, without an account.

How it works

Type a website address in the field on the HumanKey homepage or on the free check page. HumanKey reads these parts, in this order:

  1. Rules for crawlers (robots.txt): whether search engines may read the homepage, and whether the file has a Content-Signal line.
  2. Your homepage (HTML): the title, the description, structured data, headings and the text that is there before scripts run.
  3. AI guide file and list of pages (llms.txt, sitemap): whether the site offers them, and where its canonical link leads. A page on another site is not opened.

The result appears part by part while you wait, and the read path at the top shows how many things in each part are worth a look.

What you get

The result starts with one plain sentence, for example: "We checked what AI crawlers read on example.com. 2 things worth a look." Below it, the findings are grouped:

  • Worth a look: something HumanKey did not find, or could not read.
  • Could be added: optional things the site does not have yet, such as an llms.txt file or a Content-Signal line in robots.txt.
  • Good to know: neutral information.
  • Found: what is already in place. This group is folded, so it does not push the rest down.

Open a row to see the details: why it matters and, where one exists, the published rule behind it, with the date HumanKey read that rule. Text taken from the site is shown in a typewriter-style font, quoted exactly as written.

"What a crawler gets from your HTML"

One card shows your homepage's HTML as HumanKey received it, without running scripts. It quotes the page title, the description, the first headings and more, as written in the HTML. If the title you see in the browser is not in this card, a crawler that does not run scripts may not get it either.

Why there is no score

The rows describe the HTML HumanKey read. They are not a rating of the site. A single number would hide the one thing you need: which row to look at first. It would also suggest that we know how each search engine or AI system weighs these things. We do not, so we show what we found and where the rule comes from, and leave the weighing to you.

When the check stops early

The check follows the site's own rules:

  • If the site's robots.txt asks HumanKey's crawler (HumanKey-Audit) not to read the homepage, HumanKey follows that request and says so.
  • If the homepage's headers or meta tags opt out of text and data mining, HumanKey reads only its robots.txt and those headers and meta tags.

Privacy

  • You need no account and no e-mail address.
  • The address you type does not appear in your browser's address bar.
  • The check reads only public files of the address you enter and does not follow links to other sites.
  • A repeat check of the same address may show an earlier result; the date shows when it was read.

How HumanKey handles a check's data: privacy policy.

After the check: what you can set in HumanKey

Under the result, a short list shows what you can set or see in HumanKey, plan by plan, with the lines that match your result first:

  • Free
    • Choose by hand which AI crawlers may read your site (a limited number).
    • Say how AI may use your content (search, AI answers, AI training) and add a licence link.
    • Ready-made robots.txt lines built from your settings.
    • An llms.txt file built from your description.
    • These check rows for your own site.
    • With the HumanKey script installed: AI crawlers among the visits it measures on your site.
  • Pro adds
    • Fix links for the rows you can fix in your page, its head, robots.txt or settings.
    • The AI file package: robots.txt lines, an llms.txt file and tags for your homepage's head, with install steps.
    • Guided structured data.
    • Suggestions for AI crawler rules, each with a reason.
    • Visits that come from AI assistants.
  • Business adds
    • A comparison of your live robots.txt with the file HumanKey builds for your settings.
    • A weekly AI digest.
    • Where AI crawlers come from.
    • AI-written insights about your traffic.
    • A detailed report for each bot.

Compare plans for prices and limits.

Try it

Check your homepage now. It takes one address and no sign-up.

HumanKey is a service for businesses and for people acting for a business.

Know Your AI Traffic

Start tracking AI crawlers visiting your website today. Free for up to 1,000 checks per month.

Start Free Trial