← all sites
News sites vs AI crawlers
36 tracked news sites, 33 with readable policy. Snapshot: 2026-09-13
93.9%
block ≥1 AI crawler (31/33)
ClaudeBot
most-blocked bot here (27 sites)
1
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| usatoday.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| dailymail.co.uk | — | blocked | blocked | blocked | blocked | blocked | blocked |
| reuters.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| wsj.com | ? | restricted | blocked | blocked | blocked | blocked | blocked |
| cnn.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| theverge.com | — | allowed | blocked | blocked | blocked | blocked | blocked |
| vox.com | — | allowed | blocked | blocked | blocked | blocked | blocked |
| nytimes.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| arstechnica.com | — | restricted | blocked | blocked | blocked | blocked | blocked |
| huffpost.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| nbcnews.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| bloomberg.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| newyorker.com | — | restricted | blocked | blocked | blocked | blocked | blocked |
| theatlantic.com | — | allowed | blocked | blocked | blocked | blocked | blocked |
| wired.com | — | restricted | blocked | blocked | blocked | blocked | blocked |
| bbc.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| nypost.com | — | restricted | blocked | blocked | blocked | blocked | blocked |
| ft.com | ? | restricted | blocked | blocked | blocked | blocked | blocked |
| washingtonpost.com | ? | restricted | blocked | restricted | blocked | blocked | blocked |
| economist.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| forbes.com | — | blocked | blocked | restricted | blocked | blocked | blocked |
| newsweek.com | — | blocked | blocked | restricted | blocked | blocked | blocked |
| apnews.com | ? | blocked | blocked | restricted | blocked | blocked | restricted |
| techcrunch.com | — | blocked | blocked | blocked | restricted | blocked | blocked |
| theguardian.com | — | restricted | blocked | restricted | blocked | blocked | blocked |
| latimes.com | — | blocked | blocked | restricted | restricted | restricted | blocked |
| businessinsider.com | — | restricted | blocked | restricted | restricted | blocked | blocked |
| axios.com | — | restricted | restricted | restricted | restricted | blocked | blocked |
| 404media.co | — | blocked | restricted | restricted | restricted | restricted | restricted |
| cbsnews.com | — | blocked | restricted | restricted | restricted | restricted | restricted |
| semafor.com | — | blocked | restricted | restricted | restricted | restricted | restricted |
| foxnews.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| npr.org | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| politico.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| theinformation.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| time.com | ? | allowed | allowed | allowed | allowed | allowed | allowed |