The AI bots, ranked by unwelcome
Every AI-associated user agent we check, and how many of the 644 readable sites block it. Snapshot: 2026-10-04
| Bot | Operator | Purpose | Blocked by | |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 137 (21.3%) | |
| ChatGPT-User | OpenAI | fetch | 94 (14.6%) | |
| OAI-SearchBot | OpenAI | search | 77 (12%) | |
| ClaudeBot | Anthropic | training | 142 (22%) | |
| Claude-User | Anthropic | fetch | 100 (15.5%) | |
| Claude-SearchBot | Anthropic | search | 94 (14.6%) | |
| Google-Extended | training | 121 (18.8%) | ||
| Applebot-Extended | Apple | training | 123 (19.1%) | |
| PerplexityBot | Perplexity | search | 119 (18.5%) | |
| Perplexity-User | Perplexity | fetch | 103 (16%) | |
| CCBot | Common Crawl | training | 159 (24.7%) | |
| Bytespider | ByteDance | training | 161 (25%) | |
| meta-externalagent | Meta | training | 134 (20.8%) | |
| Amazonbot | Amazon | training | 117 (18.2%) | |
| DuckAssistBot | DuckDuckGo | search | 96 (14.9%) | |
| MistralAI-User | Mistral | fetch | 96 (14.9%) | |
| anthropic-ai | Anthropic | training | 117 (18.2%) | |
| Claude-Web | Anthropic | fetch | 117 (18.2%) | |
| cohere-ai | Cohere | training | 129 (20%) | |
| cohere-training-data-crawler | Cohere | training | 99 (15.4%) | |
| Diffbot | Diffbot | training | 133 (20.7%) | |
| omgili | Webz.io | training | 129 (20%) | |
| omgilibot | Webz.io | training | 114 (17.7%) | |
| YouBot | You.com | search | 115 (17.9%) | |
| ImagesiftBot | Hive | training | 115 (17.9%) | |
| Timpibot | Timpi | search | 113 (17.5%) | |
| Meta-ExternalFetcher | Meta | fetch | 94 (14.6%) | |
| AI2Bot | Allen Institute for AI | training | 99 (15.4%) | |
| DeepSeekBot | DeepSeek | training | 80 (12.4%) | |
| GrokBot | xAI | training | 67 (10.4%) | |
| PanguBot | Huawei | training | 95 (14.8%) | |
| Gemini-Deep-Research | fetch | 72 (11.2%) |
Purposes: training — collects data that may train models · search — builds an AI search index · fetch — retrieves a page live at a user's request. Sites are notably more willing to block training bots than fetch bots.