← all sites
Reference sites vs AI crawlers
18 tracked reference sites, 15 with readable policy. Snapshot: 2026-10-04
60%
block ≥1 AI crawler (9/15)
CCBot
most-blocked bot here (9 sites)
1
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| imdb.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| quora.com | — | blocked | blocked | restricted | blocked | blocked | blocked |
| wikihow.com | — | blocked | blocked | restricted | blocked | blocked | blocked |
| investopedia.com | ? | restricted | blocked | blocked | blocked | blocked | blocked |
| snopes.com | — | blocked | blocked | allowed | blocked | blocked | blocked |
| weather.com | yes | restricted | restricted | restricted | restricted | blocked | blocked |
| metacritic.com | — | blocked | restricted | restricted | restricted | blocked | restricted |
| dictionary.com | — | blocked | restricted | restricted | restricted | blocked | restricted |
| goodreads.com | — | blocked | restricted | restricted | restricted | blocked | restricted |
| accuweather.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| archive.org | — | restricted | restricted | restricted | restricted | restricted | restricted |
| britannica.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| genius.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| merriam-webster.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| rottentomatoes.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| stackoverflow.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| urbandictionary.com | — | allowed | allowed | allowed | allowed | allowed | allowed |
| wikipedia.org | — | restricted | restricted | restricted | restricted | restricted | restricted |