The AI bots, ranked by unwelcome
Every AI-associated user agent we check, and how many of the 645 readable sites block it. Snapshot: 2026-10-25
| Bot | Operator | Purpose | Blocked by | |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 139 (21.6%) | |
| ChatGPT-User | OpenAI | fetch | 93 (14.4%) | |
| OAI-SearchBot | OpenAI | search | 76 (11.8%) | |
| ClaudeBot | Anthropic | training | 144 (22.3%) | |
| Claude-User | Anthropic | fetch | 100 (15.5%) | |
| Claude-SearchBot | Anthropic | search | 94 (14.6%) | |
| Google-Extended | training | 120 (18.6%) | ||
| Applebot-Extended | Apple | training | 125 (19.4%) | |
| PerplexityBot | Perplexity | search | 118 (18.3%) | |
| Perplexity-User | Perplexity | fetch | 102 (15.8%) | |
| CCBot | Common Crawl | training | 159 (24.7%) | |
| Bytespider | ByteDance | training | 162 (25.1%) | |
| meta-externalagent | Meta | training | 133 (20.6%) | |
| Amazonbot | Amazon | training | 119 (18.4%) | |
| DuckAssistBot | DuckDuckGo | search | 95 (14.7%) | |
| MistralAI-User | Mistral | fetch | 95 (14.7%) | |
| anthropic-ai | Anthropic | training | 117 (18.1%) | |
| Claude-Web | Anthropic | fetch | 117 (18.1%) | |
| cohere-ai | Cohere | training | 129 (20%) | |
| cohere-training-data-crawler | Cohere | training | 98 (15.2%) | |
| Diffbot | Diffbot | training | 137 (21.2%) | |
| omgili | Webz.io | training | 130 (20.2%) | |
| omgilibot | Webz.io | training | 114 (17.7%) | |
| YouBot | You.com | search | 114 (17.7%) | |
| ImagesiftBot | Hive | training | 114 (17.7%) | |
| Timpibot | Timpi | search | 112 (17.4%) | |
| Meta-ExternalFetcher | Meta | fetch | 95 (14.7%) | |
| AI2Bot | Allen Institute for AI | training | 98 (15.2%) | |
| DeepSeekBot | DeepSeek | training | 79 (12.2%) | |
| GrokBot | xAI | training | 67 (10.4%) | |
| PanguBot | Huawei | training | 94 (14.6%) | |
| Gemini-Deep-Research | fetch | 71 (11%) |
Purposes: training — collects data that may train models · search — builds an AI search index · fetch — retrieves a page live at a user's request. Sites are notably more willing to block training bots than fetch bots.