← all sites
News sites vs AI crawlers
36 tracked news sites, 33 with readable policy. Snapshot: 2026-08-25
93.9%
block ≥1 AI crawler (31/33)
ClaudeBot
most-blocked bot here (27 sites)
1
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| dailymail.co.uk | — | blocked | blocked | blocked | blocked | blocked | blocked |
| usatoday.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| cnn.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| huffpost.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| nbcnews.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| bloomberg.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| nytimes.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| reuters.com | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| arstechnica.com | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| newyorker.com | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| theverge.com | — | allowed | blocked | blocked | blocked | blocked | blocked |
| vox.com | — | allowed | blocked | blocked | blocked | blocked | blocked |
| wired.com | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| wsj.com | — | restricted | blocked* | blocked* | blocked* | blocked* | blocked* |
| bbc.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| economist.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| theatlantic.com | — | allowed | blocked | blocked | blocked | blocked | blocked |
| nypost.com | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| theguardian.com | — | restricted* | blocked | restricted* | blocked | blocked | blocked |
| apnews.com | — | blocked | blocked | restricted* | blocked | blocked | restricted* |
| forbes.com | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| ft.com | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| newsweek.com | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| techcrunch.com | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| washingtonpost.com | — | restricted* | blocked | restricted* | blocked | blocked | blocked |
| axios.com | — | restricted* | restricted* | restricted | restricted* | blocked | blocked |
| businessinsider.com | — | restricted* | blocked | restricted* | restricted* | blocked | blocked |
| latimes.com | — | blocked | blocked | restricted* | restricted* | restricted* | blocked |
| 404media.co | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| cbsnews.com | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| semafor.com | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| foxnews.com | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| npr.org | — | unknown | unknown | unknown | unknown | unknown | unknown |
| politico.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| theinformation.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| time.com | — | allowed | allowed | allowed | allowed | allowed | allowed |