The AI bots, ranked by unwelcome
Every AI-associated user agent we check, and how many of the 649 readable sites block it. Snapshot: 2026-09-13
| Bot | Operator | Purpose | Blocked by | |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 141 (21.7%) | |
| ChatGPT-User | OpenAI | fetch | 96 (14.8%) | |
| OAI-SearchBot | OpenAI | search | 78 (12%) | |
| ClaudeBot | Anthropic | training | 146 (22.5%) | |
| Claude-User | Anthropic | fetch | 101 (15.6%) | |
| Claude-SearchBot | Anthropic | search | 95 (14.6%) | |
| Google-Extended | training | 126 (19.4%) | ||
| Applebot-Extended | Apple | training | 128 (19.7%) | |
| PerplexityBot | Perplexity | search | 123 (19%) | |
| Perplexity-User | Perplexity | fetch | 104 (16%) | |
| CCBot | Common Crawl | training | 164 (25.3%) | |
| Bytespider | ByteDance | training | 163 (25.1%) | |
| meta-externalagent | Meta | training | 138 (21.3%) | |
| Amazonbot | Amazon | training | 121 (18.6%) | |
| DuckAssistBot | DuckDuckGo | search | 97 (14.9%) | |
| MistralAI-User | Mistral | fetch | 97 (14.9%) | |
| anthropic-ai | Anthropic | training | 119 (18.3%) | |
| Claude-Web | Anthropic | fetch | 119 (18.3%) | |
| cohere-ai | Cohere | training | 130 (20%) | |
| cohere-training-data-crawler | Cohere | training | 100 (15.4%) | |
| Diffbot | Diffbot | training | 133 (20.5%) | |
| omgili | Webz.io | training | 130 (20%) | |
| omgilibot | Webz.io | training | 114 (17.6%) | |
| YouBot | You.com | search | 114 (17.6%) | |
| ImagesiftBot | Hive | training | 116 (17.9%) | |
| Timpibot | Timpi | search | 113 (17.4%) | |
| Meta-ExternalFetcher | Meta | fetch | 96 (14.8%) | |
| AI2Bot | Allen Institute for AI | training | 99 (15.3%) | |
| DeepSeekBot | DeepSeek | training | 81 (12.5%) | |
| GrokBot | xAI | training | 67 (10.3%) | |
| PanguBot | Huawei | training | 96 (14.8%) | |
| Gemini-Deep-Research | fetch | 73 (11.2%) |
Purposes: training — collects data that may train models · search — builds an AI search index · fetch — retrieves a page live at a user's request. Sites are notably more willing to block training bots than fetch bots.