The AI bots, ranked by unwelcome
Every AI-associated user agent we check, and how many of the 652 readable sites block it. Snapshot: 2026-08-25
| Bot | Operator | Purpose | Blocked by | |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 142 (21.8%) | |
| ChatGPT-User | OpenAI | fetch | 95 (14.6%) | |
| OAI-SearchBot | OpenAI | search | 76 (11.7%) | |
| ClaudeBot | Anthropic | training | 149 (22.9%) | |
| Claude-User | Anthropic | fetch | 99 (15.2%) | |
| Claude-SearchBot | Anthropic | search | 93 (14.3%) | |
| Google-Extended | training | 129 (19.8%) | ||
| Applebot-Extended | Apple | training | 129 (19.8%) | |
| PerplexityBot | Perplexity | search | 121 (18.6%) | |
| Perplexity-User | Perplexity | fetch | 102 (15.6%) | |
| CCBot | Common Crawl | training | 166 (25.5%) | |
| Bytespider | ByteDance | training | 163 (25%) | |
| meta-externalagent | Meta | training | 139 (21.3%) | |
| Amazonbot | Amazon | training | 120 (18.4%) | |
| DuckAssistBot | DuckDuckGo | search | 95 (14.6%) | |
| MistralAI-User | Mistral | fetch | 95 (14.6%) |
Purposes: training — collects data that may train models · search — builds an AI search index · fetch — retrieves a page live at a user's request. Sites are notably more willing to block training bots than fetch bots.