← all sites
Health sites vs AI crawlers
7 tracked health sites, 5 with readable policy. Snapshot: 2026-09-14
60%
block ≥1 AI crawler (3/5)
CCBot
most-blocked bot here (3 sites)
1
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| healthline.com | — | blocked | blocked | restricted | restricted | blocked | blocked |
| webmd.com | — | blocked | blocked | restricted | restricted | blocked | restricted |
| who.int | — | allowed | allowed | allowed | allowed | blocked | allowed |
| cdc.gov | — | restricted | restricted | restricted | restricted | restricted | restricted |
| clevelandclinic.org | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| mayoclinic.org | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| nih.gov | ? | unknown | unknown | unknown | unknown | unknown | unknown |