AI Crawlers & Bots
HumanKey detects 75+ AI crawlers in real time. Updated as new bots emerge. See exactly who is crawling your site and why.
Showing 78 of 78 bots
| Bot Name | Purpose |
|---|---|
| AhrefsBotAhrefs | SEO |
| AI2BotAllen Institute for AI | Training |
| AmazonbotAmazon | Alexa |
| Amzn-SearchBotAmazon | Search Index |
| Amzn-UserAmazon | User Query |
| Anchor BrowserAnchor | User Query |
| Claude-SearchBotAnthropic | Search |
| Claude-UserAnthropic | User Query |
| Claude-WebAnthropic | User Query |
| ClaudeBotAnthropic | Training |
| ApplebotApple | Siri |
| Applebot-ExtendedApple | Apple Intelligence |
| Arquivo-web-crawlerArquivo | Research |
| BaiduspiderBaidu | Search Index |
| BrightbotBright Data | Training |
| BytespiderByteDance | Training |
| TikTokSpiderByteDance | Training |
| TerraCottaCeramic | Search |
| cohere-aiCohere | Training |
| CCBotCommon Crawl | Training |
| DataForSeoBotDataForSEO | Search Index |
| DeepSeekBotDeepSeek | Training |
| DiffbotDiffbot | Training |
| DuckAssistBotDuckDuckGo | Search |
| DuckDuckBotDuckDuckGo | Search Index |
| AdsBot-GoogleGoogle | Advertising |
| Google-AgentGoogle | User Query |
| Google-CloudVertexBotGoogle | Training |
| Google-ExtendedGoogle | Gemini |
| Google-InspectionToolGoogle | Monitoring |
| Google-NotebookLMGoogle | User Query |
| GoogleOtherGoogle | Research |
| GoogleOther-ImageGoogle | Research |
| GoogleOther-VideoGoogle | Research |
| GooglebotGoogle | Search Index |
| Googlebot-ImageGoogle | Search Index |
| Googlebot-NewsGoogle | Search Index |
| Googlebot-VideoGoogle | Search Index |
| Storebot-GoogleGoogle | Search Index |
| PanguBotHuawei | Training |
| PetalBotHuawei | Search Index |
| ImagesiftBotImagesift | Training |
| KagiBotKagi | Search Index |
| LinkedInBotLinkedIn | Social |
| MJ12BotMajestic | SEO |
| Manus-UserManus | User Query |
| FacebookBotMeta | Training |
| Meta-ExternalAgentMeta | Training |
| BingbotMicrosoft | Search Index |
| MistralAI-UserMistral AI | User Query |
| MistralBotMistral AI | Training |
| MojeekBotMojeek | Search Index |
| ICC-CrawlerNICT (Japan) | Research |
| YetiNaver | Search Index |
| NovellumNovellum | Training |
| ChatGPT-UserOpenAI | User Query |
| GPTBotOpenAI | Training |
| OAI-AdsBotOpenAI | Advertising |
| OAI-SearchBotOpenAI | Search |
| Perplexity-UserPerplexity | User Query |
| PerplexityBotPerplexity | Search |
| ProRataIncProRata.ai | Training |
| 360SpiderQihoo 360 | Search Index |
| QwantifyQwant | Search Index |
| SEBot-WASE Ranking | SEO |
| SemrushBotSemrush | Search Index |
| SeznamBotSeznam.cz | Search Index |
| YisouSpiderShenma (Alibaba) | Search Index |
| SlackbotSlack | Social |
| SogouSpiderSogou (Tencent) | Search Index |
| TimpiBotTimpi | Training |
| UptimeRobotUptimeRobot | Monitoring |
| VelenPublicWebCrawlerVelen.io (Hunter) | Training |
| Webzio-ExtendedWebz.io | Training |
| TwitterbotX (Twitter) | Social |
| YandexBotYandex | Search Index |
| YouBotYou.com | Search |
| GrokBotxAI | Training |
This is the complete detection database. New AI crawlers are researched automatically as they emerge and added after review via our AI-powered bot research — so the list keeps pace with the AI landscape.
robots.txt compliance is inconsistent
A peer-reviewed 2025 study of large-scale AI-crawler traffic (ACM Internet Measurement Conference, IMC '25) found that robots.txt compliance drops sharply as directives get stricter, and that AI search crawlers often skip checking it entirely. Robots.txt alone can't tell you what's actually happening on your site — HumanKey's robots.txt audit (Business plans and above) checks your rules for gaps, and real-time detection, on every plan, shows you which crawlers actually ignore them.
Detect a bot we don't cover yet?
We expand the detection database as new AI crawlers emerge. Let us know which bot you're seeing and we'll look into adding it.
Smart Bot Config
Classify, allow, or block AI crawlers with one click. AI-powered recommendations based on your traffic patterns. Available on Pro plans and above.
Try in Dashboard →Start detecting all these bots on your site — free, in under 5 minutes.
Start Free Trial →No credit card required · GDPR-native · WordPress plugin coming soon