← all sites
Reference sites vs AI crawlers
18 tracked reference sites, 16 with readable policy. Snapshot: 2026-08-25
62.5%
block ≥1 AI crawler (10/16)
CCBot
most-blocked bot here (10 sites)
1
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| imdb.com | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| quora.com | — | blocked | blocked | restricted | blocked | blocked* | blocked |
| wikihow.com | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| investopedia.com | — | restricted | blocked | blocked | blocked | blocked | blocked |
| snopes.com | — | blocked | blocked | allowed | blocked | blocked | blocked |
| weather.com | yes | blocked | blocked | blocked | blocked | blocked | blocked |
| genius.com | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| dictionary.com | — | blocked | restricted* | restricted* | restricted* | blocked | restricted* |
| metacritic.com | — | blocked | restricted* | restricted* | restricted* | blocked | restricted* |
| goodreads.com | — | blocked | restricted* | restricted* | restricted* | blocked | restricted* |
| accuweather.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| archive.org | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| britannica.com | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| merriam-webster.com | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| rottentomatoes.com | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| stackoverflow.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| urbandictionary.com | — | allowed | allowed | allowed | allowed | allowed | allowed |
| wikipedia.org | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |