← all sites
Health sites vs AI crawlers
7 tracked health sites, 5 with readable policy. Snapshot: 2026-08-25
60%
block ≥1 AI crawler (3/5)
CCBot
most-blocked bot here (3 sites)
1
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| healthline.com | — | blocked | blocked | restricted* | restricted* | blocked | blocked |
| webmd.com | — | blocked | blocked | restricted* | restricted* | blocked | restricted* |
| who.int | — | allowed | allowed | allowed | allowed | blocked | allowed |
| cdc.gov | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| clevelandclinic.org | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| mayoclinic.org | — | unknown | unknown | unknown | unknown | unknown | unknown |
| nih.gov | — | unknown | unknown | unknown | unknown | unknown | unknown |