AI Crawlers & Bots

HumanKey detects 60+ AI crawlers in real time. Updated as new bots emerge. See exactly who is crawling your site and why.

60+

Bots Detected

13+

AI Companies

Real-time

Detection

GDPR

Compliant

Purpose:AI TrainingUser QueryAI SearchSearch IndexVoice AssistantApple IntelligenceAlexa / Voice Assistant

Bot Name	Company	Purpose	Description	Status
GPTBotOpenAI	OpenAI	AI Training	Web crawler for AI model training	Active
ChatGPT-UserOpenAI	OpenAI	User Query	Real-time browsing for ChatGPT conversations	Active
OAI-SearchBotOpenAI	OpenAI	AI Search	Search feature crawler	Active
ClaudeBotAnthropic	Anthropic	AI Training	Web crawler for AI model training	Active
Claude-WebAnthropic	Anthropic	User Query	Real-time browsing for Claude conversations	Active
Google-ExtendedGoogle	Google	AI Training	Opt-out crawler for AI training data	Active
GooglebotGoogle	Google	Search Index	Search indexing crawler	Active
PerplexityBotPerplexity AI	Perplexity AI	AI Search	AI-powered search engine crawler	Active
Meta-ExternalAgentMeta	Meta	AI Training	AI training data crawler	Active
FacebookBotMeta	Meta	AI Training	Content indexing and AI training crawler	Active
ApplebotApple	Apple	Voice Assistant	Siri and Spotlight suggestions crawler	Active
Applebot-ExtendedApple	Apple	Apple Intelligence	Apple Intelligence AI training crawler	Active
BingbotMicrosoft	Microsoft	Search Index	Bing search and Copilot crawler	Active
BytespiderByteDance	ByteDance	AI Training	AI model training crawler	Active
AmazonbotAmazon	Amazon	Alexa / Voice Assistant	Alexa and Amazon AI services crawler	Active
CCBotCommon Crawl	Common Crawl	AI Training	Open dataset for AI model training	Active
cohere-aiCohere	Cohere	AI Training	Enterprise AI model training crawler	Active
AI2BotAI21 Labs / Allen AI	AI21 Labs / Allen AI	AI Training	AI research crawler	Active
YouBotYou.com	You.com	AI Search	AI-enhanced search engine crawler	Active

...and more detected automatically via AI-powered bot research. New crawlers are identified and added to the database without manual updates.

⚠️ robots.txt bypass is growing

In Q1 2025, 12.9% of AI crawlers bypass robots.txt instructions (up from 3.3% in Q4 2024). Simply blocking bots in robots.txt is no longer sufficient — you need real-time detection to see what's actually happening. HumanKey's robots.txt audit checks your current rules and highlights gaps where bots may be slipping through.

Detect a bot we don't cover yet?

We add new AI crawlers continuously. Let us know which bot you're seeing and we'll add it to the detection database.

Report a Bot

Smart Bot Config

Classify, allow, or block AI crawlers with one click. AI-powered recommendations based on your traffic patterns. Available on Pro plans and above.

Try in Dashboard →

Start detecting all these bots on your site — free, in under 5 minutes.

Start Free Trial →

No credit card required · GDPR-native · WordPress plugin available