AI Crawlers & Bots
HumanKey detects 85+ AI crawlers and other bots in real time. See which of them appear among the visits measured on your site, and why they crawl.
Showing 90 of 90 bots
| Bot Name | Purpose |
|---|---|
| AhrefsBotAhrefs | SEO |
| AhrefsSiteAuditAhrefs | SEO |
| AI2BotAllen Institute for AI | Training |
| AmazonbotAmazon | Alexa |
| Amzn-SearchBotAmazon | Search Index |
| Amzn-UserAmazon | User Query |
| Anchor BrowserAnchor | User Query |
| Claude-SearchBotAnthropic | Search |
| Claude-UserAnthropic | User Query |
| Claude-WebAnthropic | User Query |
| ClaudeBotAnthropic | Training |
| ApplebotApple | Siri |
| Applebot-ExtendedApple | Apple Intelligence |
| Arquivo-web-crawlerArquivo | Research |
| BaiduspiderBaidu | Search Index |
| BitSightBotBitSight | Monitoring |
| BrightbotBright Data | Training |
| BytespiderByteDance | Training |
| TikTokSpiderByteDance | Training |
| TerraCottaCeramic | Search |
| cohere-aiCohere | Training |
| CCBotCommon Crawl | Training |
| DataForSeoBotDataForSEO | Search Index |
| DeepSeekBotDeepSeek | Training |
| DiffbotDiffbot | Training |
| DuckAssistBotDuckDuckGo | Search |
| DuckDuckBotDuckDuckGo | Search Index |
| AdsBot-GoogleGoogle | Advertising |
| Google-AgentGoogle | User Query |
| Google-CloudVertexBotGoogle | Training |
| Google-ExtendedGoogle | Gemini |
| Google-InspectionToolGoogle | Monitoring |
| Google-NotebookLMGoogle | User Query |
| Google-Read-AloudGoogle | User Query |
| GoogleOtherGoogle | Research |
| GoogleOther-ImageGoogle | Research |
| GoogleOther-VideoGoogle | Research |
| GooglebotGoogle | Search Index |
| Googlebot-ImageGoogle | Search Index |
| Googlebot-NewsGoogle | Search Index |
| Googlebot-VideoGoogle | Search Index |
| Storebot-GoogleGoogle | Search Index |
| PanguBotHuawei | Training |
| PetalBotHuawei | Search Index |
| HubSpot CrawlerHubSpot | SEO |
| ImagesiftBotImagesift | Training |
| KagiBotKagi | Search Index |
| KeenableBotKeenable | Search |
| LinkedInBotLinkedIn | Social |
| MJ12BotMajestic | SEO |
| Manus-UserManus | User Query |
| FacebookBotMeta | Training |
| Meta-ExternalAgentMeta | Training |
| meta-webindexerMeta | Search |
| BingbotMicrosoft | Search Index |
| MistralAI-UserMistral AI | User Query |
| MistralBotMistral AI | Training |
| MojeekBotMojeek | Search Index |
| ICC-CrawlerNICT (Japan) | Research |
| YetiNaver | Search Index |
| NovellumNovellum | Training |
| ChatGPT-UserOpenAI | User Query |
| GPTBotOpenAI | Training |
| OAI-AdsBotOpenAI | Advertising |
| OAI-SearchBotOpenAI | Search |
| Perplexity-UserPerplexity | User Query |
| PerplexityBotPerplexity | Search |
| ProRataIncProRata.ai | Training |
| 360SpiderQihoo 360 | Search Index |
| QwantifyQwant | Search Index |
| SEBot-WASE Ranking | SEO |
| SeekportBotSeekport | Search Index |
| SemrushBotSemrush | Search Index |
| SeznamBotSeznam.cz | Search Index |
| YisouSpiderShenma (Alibaba) | Search Index |
| SlackbotSlack | Social |
| SogouSpiderSogou (Tencent) | Search Index |
| TimpiBotTimpi | Training |
| ABEvalBotUnknown operator | Research |
| ShapBotUnknown operator | Research |
| UptimeRobotUptimeRobot | Monitoring |
| VelenPublicWebCrawlerVelen.io (Hunter) | Training |
| VocationsBotVocations | Search Index |
| Webzio-ExtendedWebz.io | Training |
| TwitterbotX (Twitter) | Social |
| YandexAccessibilityBotYandex | Monitoring |
| YandexBotYandex | Search Index |
| YandexRenderResourcesBotYandex | Search Index |
| YouBotYou.com | Search |
| GrokBotxAI | Training |
This is the full list of crawlers and bots HumanKey recognises by name. New names are added after a review, as new crawlers appear.
robots.txt compliance is inconsistent
A peer-reviewed 2025 study of large-scale AI-crawler traffic (ACM Internet Measurement Conference, IMC '25) found that robots.txt compliance drops sharply as directives get stricter, and that AI search crawlers often skip checking it entirely. Robots.txt alone can't tell you what's actually happening on your site — HumanKey's robots.txt audit (Business plans and above) checks your rules for gaps, and real-time detection, on every plan, shows which crawlers ignore them among the visits measured on your site.
Detect a bot we don't cover yet?
We expand the detection database as new AI crawlers emerge. Let us know which bot you're seeing and we'll look into adding it.
Smart Bot Config
Classify, allow, or block AI crawlers with one click. AI-powered recommendations based on your traffic patterns. Available on Pro plans and above.
Try in Dashboard →Start detecting all these bots on your site — free, in under 5 minutes.
Start Free Trial →No credit card required · GDPR-native · WordPress plugin coming soon