"Unknown bot" is not an answer: describing automated traffic that has no name

The number nobody could act on

Many traffic reports that separate people from machines end up with three groups: human visits, bots with a name (a search engine, an AI crawler, a monitoring service), and a third group that usually carries a label like "unknown bots" or "other".

The third group is often the most interesting one, and the label tells you nothing. A site owner who sees "several dozen unknown bots yesterday" cannot answer the only questions that matter: what kind of thing was this, and should I do anything about it?

We asked ourselves exactly that while reading our own reports. We did not have a good answer on the screen, although our system did know more than it was showing. So we changed what we show.

What changed in the daily report

Next to the number of automated visits without a bot name, the daily e-mail report now lists those visits by kind:

Visits from named crawlers — search engines, AI crawlers, link-preview services — are counted as before and are not part of this table.

Three rules we set for ourselves

1. A kind says what, never who. "Server or proxy network" does not name a hosting company, and "automated browser" does not name a tool. We give a visit a crawler's name only when it matches an entry in our crawler catalog; an automated visit that matches no catalog entry is described by its kind, never given a name. Anything else would be guessing presented as fact.

2. A label must be true for every visit under it. A description that fits only the typical case does not belong in a customer's report. If we cannot say something about every visit of a kind, we do not say it.

3. The leftover group stays, and stays honest. There will always be visits we cannot describe further. They are shown as "not classified further" — last in the list, never hidden inside another kind to make the table look complete.

What stays the same

• The human-or-bot verdict is computed the same way on every plan. The new table describes visits that were already counted as automated; it does not move a visit from one side to the other.
• Counts are visits, not a number of distinct bots. In our visit records, a visitor's IP address is stored only as a hash that changes every day, so we do not claim to know how many separate machines were behind the visits.
• A kind is not a rule. Blocking and allowing still work on named crawlers. A kind is a description to help you read the numbers.

What a site owner can do with it

• A rise in "server or proxy network" visits is a reason to check in your own server logs which pages they reach. Pages with no public value (login, search results, cart) are worth a look first, together with your rate limits.
• A rise in "automated browser" visits can mean that someone is collecting your content or trying out a form. Your own server logs show which pages were involved.
• "Self-declared bot" visits are the polite ones: they say what they are. They are candidates for our catalog of named crawlers.
• A large "not classified further" share is a signal for us, not for you. That number is how we judge this feature.

Questions about how to read your report? Write to us through the contact form.