AI Crawlers & Bots
HumanKey detects 60+ AI crawlers in real time. Updated as new bots emerge. See exactly who is crawling your site and why.
Showing 63 of 63 bots
| Bot Name | Purpose |
|---|---|
| AhrefsBotAhrefs | SEO |
| AI2BotAllen Institute for AI | Training |
| AmazonbotAmazon | Alexa |
| Claude-SearchBotAnthropic | Search |
| Claude-UserAnthropic | User Query |
| Claude-WebAnthropic | User Query |
| ClaudeBotAnthropic | Training |
| ApplebotApple | Siri |
| Applebot-ExtendedApple | Apple Intelligence |
| BaiduspiderBaidu | Search Index |
| BytespiderByteDance | Training |
| cohere-aiCohere | Training |
| CCBotCommon Crawl | Training |
| DataForSeoBotDataForSEO | Search Index |
| DeepSeekBotDeepSeek | Training |
| DiffbotDiffbot | Training |
| DuckAssistBotDuckDuckGo | Search |
| DuckDuckBotDuckDuckGo | Search Index |
| AdsBot-GoogleGoogle | Advertising |
| Google-CloudVertexBotGoogle | Training |
| Google-ExtendedGoogle | Gemini |
| Google-InspectionToolGoogle | Monitoring |
| GoogleOtherGoogle | Research |
| GooglebotGoogle | Search Index |
| Googlebot-ImageGoogle | Search Index |
| Googlebot-NewsGoogle | Search Index |
| Googlebot-VideoGoogle | Search Index |
| Storebot-GoogleGoogle | Search Index |
| PanguBotHuawei | Training |
| PetalBotHuawei | Search Index |
| ImagesiftBotImagesift | Training |
| KagiBotKagi | Search Index |
| LinkedInBotLinkedIn | Social |
| MJ12BotMajestic | SEO |
| FacebookBotMeta | Training |
| Meta-ExternalAgentMeta | Training |
| BingbotMicrosoft | Search Index |
| MistralAI-UserMistral AI | User Query |
| MistralBotMistral AI | Training |
| MojeekBotMojeek | Search Index |
| ICC-CrawlerNICT (Japan) | Research |
| YetiNaver | Search Index |
| ChatGPT-UserOpenAI | User Query |
| GPTBotOpenAI | Training |
| OAI-AdsBotOpenAI | Advertising |
| OAI-SearchBotOpenAI | Search |
| Perplexity-UserPerplexity | User Query |
| PerplexityBotPerplexity | Search |
| 360SpiderQihoo 360 | Search Index |
| QwantifyQwant | Search Index |
| SemrushBotSemrush | Search Index |
| SeznamBotSeznam.cz | Search Index |
| YisouSpiderShenma (Alibaba) | Search Index |
| SlackbotSlack | Social |
| SogouSpiderSogou (Tencent) | Search Index |
| TimpiBotTimpi | Training |
| UptimeRobotUptimeRobot | Monitoring |
| VelenPublicWebCrawlerVelen.io (Hunter) | Training |
| Webzio-ExtendedWebz.io | Training |
| TwitterbotX (Twitter) | Social |
| YandexBotYandex | Search Index |
| YouBotYou.com | Search |
| GrokBotxAI | Training |
This is the complete detection database. New AI crawlers are researched automatically as they emerge and added after review via our AI-powered bot research — so the list keeps pace with the AI landscape.
robots.txt bypass is growing
In Q1 2025, 12.9% of AI crawlers bypass robots.txtinstructions (up from 3.3% in Q4 2024). Simply blocking bots in robots.txt is no longer sufficient — you need real-time detection to see what's actually happening. HumanKey's robots.txt audit (available on Business plans and above) checks your current rules and highlights gaps where bots may be slipping through — and on every plan, real-time detection shows you which crawlers actually ignore those rules.
Detect a bot we don't cover yet?
We expand the detection database as new AI crawlers emerge. Let us know which bot you're seeing and we'll look into adding it.
Smart Bot Config
Classify, allow, or block AI crawlers with one click. AI-powered recommendations based on your traffic patterns. Available on Pro plans and above.
Try in Dashboard →Start detecting all these bots on your site — free, in under 5 minutes.
Start Free Trial →No credit card required · GDPR-native · WordPress plugin coming soon