The AI bots, ranked by unwelcome
Every AI-associated user agent we check, and how many of the 646 readable sites block it. Snapshot: 2026-11-15
| Bot | Operator | Purpose | Blocked by | |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 142 (22%) | |
| ChatGPT-User | OpenAI | fetch | 92 (14.2%) | |
| OAI-SearchBot | OpenAI | search | 77 (11.9%) | |
| ClaudeBot | Anthropic | training | 149 (23.1%) | |
| Claude-User | Anthropic | fetch | 102 (15.8%) | |
| Claude-SearchBot | Anthropic | search | 99 (15.3%) | |
| Google-Extended | training | 122 (18.9%) | ||
| Applebot-Extended | Apple | training | 131 (20.3%) | |
| PerplexityBot | Perplexity | search | 120 (18.6%) | |
| Perplexity-User | Perplexity | fetch | 107 (16.6%) | |
| CCBot | Common Crawl | training | 163 (25.2%) | |
| Bytespider | ByteDance | training | 166 (25.7%) | |
| meta-externalagent | Meta | training | 138 (21.4%) | |
| Amazonbot | Amazon | training | 122 (18.9%) | |
| DuckAssistBot | DuckDuckGo | search | 98 (15.2%) | |
| MistralAI-User | Mistral | fetch | 97 (15%) | |
| anthropic-ai | Anthropic | training | 121 (18.7%) | |
| Claude-Web | Anthropic | fetch | 119 (18.4%) | |
| cohere-ai | Cohere | training | 133 (20.6%) | |
| cohere-training-data-crawler | Cohere | training | 102 (15.8%) | |
| Diffbot | Diffbot | training | 141 (21.8%) | |
| omgili | Webz.io | training | 135 (20.9%) | |
| omgilibot | Webz.io | training | 117 (18.1%) | |
| YouBot | You.com | search | 114 (17.6%) | |
| ImagesiftBot | Hive | training | 116 (18%) | |
| Timpibot | Timpi | search | 114 (17.6%) | |
| Meta-ExternalFetcher | Meta | fetch | 98 (15.2%) | |
| AI2Bot | Allen Institute for AI | training | 101 (15.6%) | |
| DeepSeekBot | DeepSeek | training | 81 (12.5%) | |
| GrokBot | xAI | training | 70 (10.8%) | |
| PanguBot | Huawei | training | 96 (14.9%) | |
| Gemini-Deep-Research | fetch | 72 (11.1%) |
Purposes: training — collects data that may train models · search — builds an AI search index · fetch — retrieves a page live at a user's request. Sites are notably more willing to block training bots than fetch bots.